Re: [openstack-dev] Following the new PTI for document build, broken local builds
On Thu, 2018-04-26 at 15:49 +0100, Stephen Finucane wrote: > On Wed, 2018-04-25 at 12:06 -0500, Sean McGinnis wrote: > > > > > > > > > > [1] https://review.openstack.org/#/c/564232/ > > > > > > > > > > > > > The only concern I have is that it may slow the transition to the > > > > python 3 version of the jobs, since someone would have to actually > > > > fix the warnings before they could add the new job. I'm not sure I > > > > want to couple the tasks of fixing doc build warnings with also > > > > making those docs build under python 3 (which is usually quite > > > > simple). > > > > > > > > Is there some other way to enable this flag independently of the move to > > > > the python3 job? > > > > > > The existing proposal is: > > > > > > https://review.openstack.org/559348 > > > > > > TL;DR if you still have a build_sphinx section in setup.cfg then defaults > > > will remain the same, but when removing it as part of the transition to > > > the > > > new PTI you'll have to eliminate any warnings. (Although AFAICT it doesn't > > > hurt to leave that section in place as long as you need, and you can still > > > do the rest of the PTI conversion.) > > > > > > The hold-up is that the job in question is also potentially used by other > > > Zuul users outside of OpenStack - including those who aren't using pbr at > > > all (i.e. there's no setup.cfg). So we need to warn those folks to > > > prepare. > > > > > > cheers, > > > Zane. > > > > > > > Ah, I had looked but did not find an existing proposal. Looks like that > > would > > work too. I am good either way, but I will leave my approach out there just > > as > > another option to consider. I'll abandon that if folks prefer this way. > > Yeah, I reviewed your patch but assumed you'd seen mine already and > were looking for a quicker alternative. > > I've started the process of adding this to zuul-jobs by posting the > warning to zuul-announce (though it's waiting moderation by corvus). We > only need to wait two weeks after sending that message before we can > merge the patch to zuul-jobs, so I guess we should go that way now? > > Stephen This is now merged: https://review.openstack.org/#/c/559348/ For anyone that has migrated to the new PTI for documentation (namely, not using 'python setup.py build_sphinx'), you may find your docs builds are now broken. This will likely be a result of regressions introduced since the migration to the new PTI and should be trivial to resolve. If you have any issues, feel free to jump on #openstack-doc where one of the lovely docs people will be happy to help. Cheers, Stephen PS: I do realize this has likely incurred some extra work for some projects. No one likes busy work and I'm sorry to have forced that upon people but this change was essential to ensure every project could rely on the gate to actually validate their documentation. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Following the new PTI for document build, broken local builds
On Wed, 2018-04-25 at 12:06 -0500, Sean McGinnis wrote: > > > > > > > > [1] https://review.openstack.org/#/c/564232/ > > > > > > > > > > The only concern I have is that it may slow the transition to the > > > python 3 version of the jobs, since someone would have to actually > > > fix the warnings before they could add the new job. I'm not sure I > > > want to couple the tasks of fixing doc build warnings with also > > > making those docs build under python 3 (which is usually quite > > > simple). > > > > > > Is there some other way to enable this flag independently of the move to > > > the python3 job? > > > > The existing proposal is: > > > > https://review.openstack.org/559348 > > > > TL;DR if you still have a build_sphinx section in setup.cfg then defaults > > will remain the same, but when removing it as part of the transition to the > > new PTI you'll have to eliminate any warnings. (Although AFAICT it doesn't > > hurt to leave that section in place as long as you need, and you can still > > do the rest of the PTI conversion.) > > > > The hold-up is that the job in question is also potentially used by other > > Zuul users outside of OpenStack - including those who aren't using pbr at > > all (i.e. there's no setup.cfg). So we need to warn those folks to prepare. > > > > cheers, > > Zane. > > > > Ah, I had looked but did not find an existing proposal. Looks like that would > work too. I am good either way, but I will leave my approach out there just as > another option to consider. I'll abandon that if folks prefer this way. Yeah, I reviewed your patch but assumed you'd seen mine already and were looking for a quicker alternative. I've started the process of adding this to zuul-jobs by posting the warning to zuul-announce (though it's waiting moderation by corvus). We only need to wait two weeks after sending that message before we can merge the patch to zuul-jobs, so I guess we should go that way now? Stephen __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Following the new PTI for document build, broken local builds
On 2018-04-25 13:10:42 -0400 (-0400), Doug Hellmann wrote: [...] > Transitioning jobs is a bit painful because the job definitions and > job templates are defined in separately places. If we don't want > this setting to be controlled from a file within the git repo, I > guess the most expedient thing is to go ahead and make this part > of the python 3 job transition. We could provide an experimental job or some basic instructions in an announcement and just schedule a flag day to start enforcing everywhere. -- Jeremy Stanley signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Following the new PTI for document build, broken local builds
Excerpts from Sean McGinnis's message of 2018-04-25 11:55:43 -0500: > > > > > > [1] https://review.openstack.org/#/c/564232/ > > > > > > > The only concern I have is that it may slow the transition to the > > python 3 version of the jobs, since someone would have to actually > > fix the warnings before they could add the new job. I'm not sure I > > want to couple the tasks of fixing doc build warnings with also > > making those docs build under python 3 (which is usually quite > > simple). > > > > Is there some other way to enable this flag independently of the move to > > the python3 job? > > > > Doug > > > > I did consider just creating a whole new job definition. I could probably do > that instead, but my hope was those proactive enough to be moving to python 3 > to run their jobs would also be proactive enough that they have already > addressed doc job warnings. > > We could do two separate jobs, then when everyone is ready, collapse it back > to > one job. I was hoping to jump ahead a little though. > Transitioning jobs is a bit painful because the job definitions and job templates are defined in separately places. If we don't want this setting to be controlled from a file within the git repo, I guess the most expedient thing is to go ahead and make this part of the python 3 job transition. Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Following the new PTI for document build, broken local builds
> >> > >>[1] https://review.openstack.org/#/c/564232/ > >> > > > >The only concern I have is that it may slow the transition to the > >python 3 version of the jobs, since someone would have to actually > >fix the warnings before they could add the new job. I'm not sure I > >want to couple the tasks of fixing doc build warnings with also > >making those docs build under python 3 (which is usually quite > >simple). > > > >Is there some other way to enable this flag independently of the move to > >the python3 job? > > The existing proposal is: > > https://review.openstack.org/559348 > > TL;DR if you still have a build_sphinx section in setup.cfg then defaults > will remain the same, but when removing it as part of the transition to the > new PTI you'll have to eliminate any warnings. (Although AFAICT it doesn't > hurt to leave that section in place as long as you need, and you can still > do the rest of the PTI conversion.) > > The hold-up is that the job in question is also potentially used by other > Zuul users outside of OpenStack - including those who aren't using pbr at > all (i.e. there's no setup.cfg). So we need to warn those folks to prepare. > > cheers, > Zane. > Ah, I had looked but did not find an existing proposal. Looks like that would work too. I am good either way, but I will leave my approach out there just as another option to consider. I'll abandon that if folks prefer this way. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Following the new PTI for document build, broken local builds
On 25/04/18 11:41, Doug Hellmann wrote: Excerpts from Sean McGinnis's message of 2018-04-25 09:59:13 -0500: I'd be more in favour of changing the zuul job to build with the '-W' flag. To be honest, there is no good reason to not have this flag enabled. I'm not sure that will be a popular opinion though as it may break some projects' builds (correctly, but still). I'll propose a patch against zuul-jobs and see what happens :) Stephen I am in favor of this too. We will probably need to give some teams some time to get warnings fixed though. I haven't done any kind of extensive audit of projects, but from a few I looked through, there are definitely a few that are not erroring on warnings and are likely to be blocked if we suddenly flipped the switch and errored on those. This is a legitimate issue though. In Cinder we had -W in the tox docs job, but since that is no longer being enforced in the gate, running "tox -e docs" from a fresh clone of master was failing. We really do need some way to enforce this so things like that do not happen. This. While forcing work on teams to do busywork is undeniably A Very Bad Thing (TM), I do think the longer we leave this, the worse it'll get. The zuul-jobs [1] patch will probably introduce some pain for projects but it seems like inevitable pain and we're in the right part of the cycle in which to do something like this. I'd be willing to help projects fix issues they encounter, which I expect will be minimal for most projects. I too think enforcing -W is the way to go, so count me in for the broken docs build help. Thanks for pushing this forward! Cheers, pk In support of this I have proposed [1]. To make it easier to transition (since I'm pretty sure this will involve a lot of work by some projects) and since we want to eventually have everything run under Python 3, I have just proposed setting this flag as the default for the publish-openstack-sphinx-docs-python3 job template. Then projects can opt in as they are ready for both the warnings-as-errors and Python 3 support. I would love to hear if there are any concerns about doing things this way or if anyone has any better suggestions. Thanks! Sean [1] https://review.openstack.org/#/c/564232/ The only concern I have is that it may slow the transition to the python 3 version of the jobs, since someone would have to actually fix the warnings before they could add the new job. I'm not sure I want to couple the tasks of fixing doc build warnings with also making those docs build under python 3 (which is usually quite simple). Is there some other way to enable this flag independently of the move to the python3 job? The existing proposal is: https://review.openstack.org/559348 TL;DR if you still have a build_sphinx section in setup.cfg then defaults will remain the same, but when removing it as part of the transition to the new PTI you'll have to eliminate any warnings. (Although AFAICT it doesn't hurt to leave that section in place as long as you need, and you can still do the rest of the PTI conversion.) The hold-up is that the job in question is also potentially used by other Zuul users outside of OpenStack - including those who aren't using pbr at all (i.e. there's no setup.cfg). So we need to warn those folks to prepare. cheers, Zane. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Following the new PTI for document build, broken local builds
> > > > [1] https://review.openstack.org/#/c/564232/ > > > > The only concern I have is that it may slow the transition to the > python 3 version of the jobs, since someone would have to actually > fix the warnings before they could add the new job. I'm not sure I > want to couple the tasks of fixing doc build warnings with also > making those docs build under python 3 (which is usually quite > simple). > > Is there some other way to enable this flag independently of the move to > the python3 job? > > Doug > I did consider just creating a whole new job definition. I could probably do that instead, but my hope was those proactive enough to be moving to python 3 to run their jobs would also be proactive enough that they have already addressed doc job warnings. We could do two separate jobs, then when everyone is ready, collapse it back to one job. I was hoping to jump ahead a little though. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Following the new PTI for document build, broken local builds
Excerpts from Sean McGinnis's message of 2018-04-25 09:59:13 -0500: > > > > > > > > > > I'd be more in favour of changing the zuul job to build with the '-W' > > > > > flag. To be honest, there is no good reason to not have this flag > > > > > enabled. I'm not sure that will be a popular opinion though as it may > > > > > break some projects' builds (correctly, but still). > > > > > > > > > > I'll propose a patch against zuul-jobs and see what happens :) > > > > > > > > > > Stephen > > > > > > > > > > > > > I am in favor of this too. We will probably need to give some teams > > > > some time > > > > to get warnings fixed though. I haven't done any kind of extensive > > > > audit of > > > > projects, but from a few I looked through, there are definitely a few > > > > that are > > > > not erroring on warnings and are likely to be blocked if we suddenly > > > > flipped > > > > the switch and errored on those. > > > > > > > > This is a legitimate issue though. In Cinder we had -W in the tox docs > > > > job, but > > > > since that is no longer being enforced in the gate, running "tox -e > > > > docs" from > > > > a fresh clone of master was failing. We really do need some way to > > > > enforce this > > > > so things like that do not happen. > > > > > > This. While forcing work on teams to do busywork is undeniably A Very > > > Bad Thing (TM), I do think the longer we leave this, the worse it'll > > > get. The zuul-jobs [1] patch will probably introduce some pain for > > > projects but it seems like inevitable pain and we're in the right part > > > of the cycle in which to do something like this. I'd be willing to help > > > projects fix issues they encounter, which I expect will be minimal for > > > most projects. > > > > I too think enforcing -W is the way to go, so count me in for the > > broken docs build help. > > > > Thanks for pushing this forward! > > > > Cheers, > > pk > > > > In support of this I have proposed [1]. To make it easier to transition (since > I'm pretty sure this will involve a lot of work by some projects) and since we > want to eventually have everything run under Python 3, I have just proposed > setting this flag as the default for the publish-openstack-sphinx-docs-python3 > job template. Then projects can opt in as they are ready for both the > warnings-as-errors and Python 3 support. > > I would love to hear if there are any concerns about doing things this way or > if anyone has any better suggestions. > > Thanks! > Sean > > [1] https://review.openstack.org/#/c/564232/ > The only concern I have is that it may slow the transition to the python 3 version of the jobs, since someone would have to actually fix the warnings before they could add the new job. I'm not sure I want to couple the tasks of fixing doc build warnings with also making those docs build under python 3 (which is usually quite simple). Is there some other way to enable this flag independently of the move to the python3 job? Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Following the new PTI for document build, broken local builds
> > > > > > > > I'd be more in favour of changing the zuul job to build with the '-W' > > > > flag. To be honest, there is no good reason to not have this flag > > > > enabled. I'm not sure that will be a popular opinion though as it may > > > > break some projects' builds (correctly, but still). > > > > > > > > I'll propose a patch against zuul-jobs and see what happens :) > > > > > > > > Stephen > > > > > > > > > > I am in favor of this too. We will probably need to give some teams some > > > time > > > to get warnings fixed though. I haven't done any kind of extensive audit > > > of > > > projects, but from a few I looked through, there are definitely a few > > > that are > > > not erroring on warnings and are likely to be blocked if we suddenly > > > flipped > > > the switch and errored on those. > > > > > > This is a legitimate issue though. In Cinder we had -W in the tox docs > > > job, but > > > since that is no longer being enforced in the gate, running "tox -e docs" > > > from > > > a fresh clone of master was failing. We really do need some way to > > > enforce this > > > so things like that do not happen. > > > > This. While forcing work on teams to do busywork is undeniably A Very > > Bad Thing (TM), I do think the longer we leave this, the worse it'll > > get. The zuul-jobs [1] patch will probably introduce some pain for > > projects but it seems like inevitable pain and we're in the right part > > of the cycle in which to do something like this. I'd be willing to help > > projects fix issues they encounter, which I expect will be minimal for > > most projects. > > I too think enforcing -W is the way to go, so count me in for the > broken docs build help. > > Thanks for pushing this forward! > > Cheers, > pk > In support of this I have proposed [1]. To make it easier to transition (since I'm pretty sure this will involve a lot of work by some projects) and since we want to eventually have everything run under Python 3, I have just proposed setting this flag as the default for the publish-openstack-sphinx-docs-python3 job template. Then projects can opt in as they are ready for both the warnings-as-errors and Python 3 support. I would love to hear if there are any concerns about doing things this way or if anyone has any better suggestions. Thanks! Sean [1] https://review.openstack.org/#/c/564232/ __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Following the new PTI for document build, broken local builds
On Fri, 06 Apr 2018 15:52:46 +0100 Stephen Finucane wrote: > On Fri, 2018-04-06 at 08:02 -0500, Sean McGinnis wrote: > > > > > > > > How can we enable warning_is_error in the gate with the new PTI? It's > > > > easy enough to add the -W flag in tox.ini for local builds, but as you > > > > say the tox job is never called in the gate. In the gate zuul checks > > > > for > > > > it in the [build_sphinx] section of setup.cfg: > > > > > > > > https://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/sphinx/library/sphinx_check_warning_is_error.pyLovel#n23 > > > > > > > > [...] > > > > > > I'd be more in favour of changing the zuul job to build with the '-W' > > > flag. To be honest, there is no good reason to not have this flag > > > enabled. I'm not sure that will be a popular opinion though as it may > > > break some projects' builds (correctly, but still). > > > > > > I'll propose a patch against zuul-jobs and see what happens :) > > > > > > Stephen > > > > > > > I am in favor of this too. We will probably need to give some teams some > > time > > to get warnings fixed though. I haven't done any kind of extensive audit of > > projects, but from a few I looked through, there are definitely a few that > > are > > not erroring on warnings and are likely to be blocked if we suddenly flipped > > the switch and errored on those. > > > > This is a legitimate issue though. In Cinder we had -W in the tox docs job, > > but > > since that is no longer being enforced in the gate, running "tox -e docs" > > from > > a fresh clone of master was failing. We really do need some way to enforce > > this > > so things like that do not happen. > > This. While forcing work on teams to do busywork is undeniably A Very > Bad Thing (TM), I do think the longer we leave this, the worse it'll > get. The zuul-jobs [1] patch will probably introduce some pain for > projects but it seems like inevitable pain and we're in the right part > of the cycle in which to do something like this. I'd be willing to help > projects fix issues they encounter, which I expect will be minimal for > most projects. I too think enforcing -W is the way to go, so count me in for the broken docs build help. Thanks for pushing this forward! Cheers, pk __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Following the new PTI for document build, broken local builds
On Fri, 2018-04-06 at 08:02 -0500, Sean McGinnis wrote: > > > > > > How can we enable warning_is_error in the gate with the new PTI? It's > > > easy enough to add the -W flag in tox.ini for local builds, but as you > > > say the tox job is never called in the gate. In the gate zuul checks for > > > it in the [build_sphinx] section of setup.cfg: > > > > > > https://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/sphinx/library/sphinx_check_warning_is_error.pyLovel#n23 > > > > > > [...] > > > > I'd be more in favour of changing the zuul job to build with the '-W' > > flag. To be honest, there is no good reason to not have this flag > > enabled. I'm not sure that will be a popular opinion though as it may > > break some projects' builds (correctly, but still). > > > > I'll propose a patch against zuul-jobs and see what happens :) > > > > Stephen > > > > I am in favor of this too. We will probably need to give some teams some time > to get warnings fixed though. I haven't done any kind of extensive audit of > projects, but from a few I looked through, there are definitely a few that are > not erroring on warnings and are likely to be blocked if we suddenly flipped > the switch and errored on those. > > This is a legitimate issue though. In Cinder we had -W in the tox docs job, > but > since that is no longer being enforced in the gate, running "tox -e docs" from > a fresh clone of master was failing. We really do need some way to enforce > this > so things like that do not happen. This. While forcing work on teams to do busywork is undeniably A Very Bad Thing (TM), I do think the longer we leave this, the worse it'll get. The zuul-jobs [1] patch will probably introduce some pain for projects but it seems like inevitable pain and we're in the right part of the cycle in which to do something like this. I'd be willing to help projects fix issues they encounter, which I expect will be minimal for most projects. Stephen [1] https://review.openstack.org/559348 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Following the new PTI for document build, broken local builds
> > > > How can we enable warning_is_error in the gate with the new PTI? It's > > easy enough to add the -W flag in tox.ini for local builds, but as you > > say the tox job is never called in the gate. In the gate zuul checks for > > it in the [build_sphinx] section of setup.cfg: > > > > https://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/sphinx/library/sphinx_check_warning_is_error.py#n23 > > > > [...] > > I'd be more in favour of changing the zuul job to build with the '-W' > flag. To be honest, there is no good reason to not have this flag > enabled. I'm not sure that will be a popular opinion though as it may > break some projects' builds (correctly, but still). > > I'll propose a patch against zuul-jobs and see what happens :) > > Stephen > I am in favor of this too. We will probably need to give some teams some time to get warnings fixed though. I haven't done any kind of extensive audit of projects, but from a few I looked through, there are definitely a few that are not erroring on warnings and are likely to be blocked if we suddenly flipped the switch and errored on those. This is a legitimate issue though. In Cinder we had -W in the tox docs job, but since that is no longer being enforced in the gate, running "tox -e docs" from a fresh clone of master was failing. We really do need some way to enforce this so things like that do not happen. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Following the new PTI for document build, broken local builds
On Thu, 2018-04-05 at 16:36 -0400, Zane Bitter wrote: > On 21/03/18 06:49, Stephen Finucane wrote: > > As noted by Monty in a prior openstack-dev post [2], some projects rely > > on a pbr extension to the 'build_sphinx' setuptools command which can > > automatically run the 'sphinx-apidoc' tool before building docs. This > > is enabled by configuring some settings in the '[pbr]' section of the > > 'setup.cfg' file [3]. To ensure this continued working, the zuul jobs > > definitions [4] check for the presence of these settings and build docs > > using the legacy 'build_sphinx' command if found. **At no point do the > > jobs call the tox job**. As a result, if you convert a project to use > > 'sphinx-build' in 'tox.ini' without resolving the autodoc issues, you > > lose the ability to build docs locally. > > > > I've gone through and proposed a couple of reverts to fix projects > > we've already broken. However, going forward, there are two things > > people should do to prevent issues like this popping up. > > > > * Firstly, you should remove the '[build_sphinx]' and '[pbr]' sections > > from 'setup.cfg' in any patches that aim to convert a project to use > > the new PTI. This will ensure the gate catches any potential > > issues. > > How can we enable warning_is_error in the gate with the new PTI? It's > easy enough to add the -W flag in tox.ini for local builds, but as you > say the tox job is never called in the gate. In the gate zuul checks for > it in the [build_sphinx] section of setup.cfg: > > https://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/sphinx/library/sphinx_check_warning_is_error.py#n23 > > So I think it makes more sense to remove the [pbr] section, but leave > the [build_sphinx] section? > > thanks, > Zane. I'd be more in favour of changing the zuul job to build with the '-W' flag. To be honest, there is no good reason to not have this flag enabled. I'm not sure that will be a popular opinion though as it may break some projects' builds (correctly, but still). I'll propose a patch against zuul-jobs and see what happens :) Stephen __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Following the new PTI for document build, broken local builds
On 21/03/18 06:49, Stephen Finucane wrote: As noted by Monty in a prior openstack-dev post [2], some projects rely on a pbr extension to the 'build_sphinx' setuptools command which can automatically run the 'sphinx-apidoc' tool before building docs. This is enabled by configuring some settings in the '[pbr]' section of the 'setup.cfg' file [3]. To ensure this continued working, the zuul jobs definitions [4] check for the presence of these settings and build docs using the legacy 'build_sphinx' command if found. **At no point do the jobs call the tox job**. As a result, if you convert a project to use 'sphinx-build' in 'tox.ini' without resolving the autodoc issues, you lose the ability to build docs locally. I've gone through and proposed a couple of reverts to fix projects we've already broken. However, going forward, there are two things people should do to prevent issues like this popping up. * Firstly, you should remove the '[build_sphinx]' and '[pbr]' sections from 'setup.cfg' in any patches that aim to convert a project to use the new PTI. This will ensure the gate catches any potential issues. How can we enable warning_is_error in the gate with the new PTI? It's easy enough to add the -W flag in tox.ini for local builds, but as you say the tox job is never called in the gate. In the gate zuul checks for it in the [build_sphinx] section of setup.cfg: https://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/sphinx/library/sphinx_check_warning_is_error.py#n23 So I think it makes more sense to remove the [pbr] section, but leave the [build_sphinx] section? thanks, Zane. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Following the new PTI for document build, broken local builds
> > tl;dr: You don't _have_ to automate this stuff, but it helps. > > sphinx-apidoc generates stub files containing a whole load of autodoc > directives. As noted above, you can check the output of a sphinx-apidoc > run and you'll see just this. If I were to guess, Cinder simply checked > in the output of such a run [*] into Git, meaning they don't need to > run it each time. This works but it comes with the downside that your > docs can and will get out of sync with the actual code when, for > example, you either add or remove some modules or functions. Running > sphinx-apidoc on each build, as we've been doing with pbr's autodoc > feature, ensures this out-of-sync issue doesn't happen, at the expense > of increased doc build times. > > Stephen > > [*] They might also have handwritten this stuff, but I highly doubt > that (it's rather tedious to write). > Ah, perfect. This was the motivation I was looking for. I don't think anyone on the team is aware of this. In this case, switching over to using the new, dynamically generated way has a lot of benefit. Looking deeper, there are some custom things being done to generate this output. So rather than maintaining (or really not maintaining as is the reality here) this custom code, we should streamline this and be consistent by following the new approach. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Following the new PTI for document build, broken local builds
On Thu, 2018-03-29 at 07:47 -0500, Sean McGinnis wrote: > > > > > > It's not mentioned here, but I discovered today that Cinder is using the > > > sphinx.ext.autodoc module. Is there any issue with using this? > > > > > > > Nope - sphinx-apidoc and the likes use autodoc under the hood. You can > > see this by checking the output in 'contributor/api' or the likes. > > > > Stephen > > > > I'm wondering if there is a problem with using this vs. the way being > proposed. > > In other words, do we need to switch over to this new sphinxcontrib module, or > staying with autodoc should be OK. And if so, why not switch current users of > the pbr method over to use sphinx.ext.autdoc rather than introducing something > new? tl;dr: You don't _have_ to automate this stuff, but it helps. sphinx-apidoc generates stub files containing a whole load of autodoc directives. As noted above, you can check the output of a sphinx-apidoc run and you'll see just this. If I were to guess, Cinder simply checked in the output of such a run [*] into Git, meaning they don't need to run it each time. This works but it comes with the downside that your docs can and will get out of sync with the actual code when, for example, you either add or remove some modules or functions. Running sphinx-apidoc on each build, as we've been doing with pbr's autodoc feature, ensures this out-of-sync issue doesn't happen, at the expense of increased doc build times. Stephen [*] They might also have handwritten this stuff, but I highly doubt that (it's rather tedious to write). __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Following the new PTI for document build, broken local builds
> > > > It's not mentioned here, but I discovered today that Cinder is using the > > sphinx.ext.autodoc module. Is there any issue with using this? > > > > Nope - sphinx-apidoc and the likes use autodoc under the hood. You can > see this by checking the output in 'contributor/api' or the likes. > > Stephen > I'm wondering if there is a problem with using this vs. the way being proposed. In other words, do we need to switch over to this new sphinxcontrib module, or staying with autodoc should be OK. And if so, why not switch current users of the pbr method over to use sphinx.ext.autdoc rather than introducing something new? __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Following the new PTI for document build, broken local builds
On Wed, 2018-03-28 at 14:14 -0500, Sean McGinnis wrote: > On Thu, Mar 22, 2018 at 10:43:45AM +, Stephen Finucane wrote: > > On Wed, 2018-03-21 at 09:57 -0500, Sean McGinnis wrote: > > > On Wed, Mar 21, 2018 at 10:49:02AM +, Stephen Finucane wrote: > > > > tl;dr: Make sure you stop using pbr's autodoc feature before converting > > > > them to the new PTI for docs. > > > > > > > > [snip] > > > > > > > > That's unfortunate. What we really need is a migration path from the > > 'pbr' way of doing things to something else. I see three possible > > avenues at this point in time: > > > >1. Start using 'sphinx.ext.autosummary'. Apparently this can do similar > > things to 'sphinx-apidoc' but it takes the form of an extension. > > From my brief experiments, the output generated from this is > > radically different and far less comprehensive than what 'sphinx- > > apidoc' generates. However, it supports templating so we could > > probably configure this somehow and add our own special directive > > somewhere like 'openstackdocstheme' > >2. Push for the 'sphinx.ext.apidoc' extension I proposed some time back > > against upstream Sphinx [1]. This essentially does what the PBR > > extension does but moves configuration into 'conf.py'. However, this > > is currently held up as I can't adequately explain the differences > > between this and 'sphinx.ext.autosummary' (there's definite overlap > > but I don't understand 'autosummary' well enough to compare them). > >3. Modify the upstream jobs that detect the pbr integration and have > > them run 'sphinx-apidoc' before 'sphinx-build'. This is the least > > technically appealing approach as it still leaves us unable to build > > stuff locally and adds yet more "magic" to the gate, but it does let > > us progress. > > > > It's not mentioned here, but I discovered today that Cinder is using the > sphinx.ext.autodoc module. Is there any issue with using this? > Nope - sphinx-apidoc and the likes use autodoc under the hood. You can see this by checking the output in 'contributor/api' or the likes. Stephen > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Following the new PTI for document build, broken local builds
On Thu, Mar 22, 2018 at 10:43:45AM +, Stephen Finucane wrote: > On Wed, 2018-03-21 at 09:57 -0500, Sean McGinnis wrote: > > On Wed, Mar 21, 2018 at 10:49:02AM +, Stephen Finucane wrote: > > > tl;dr: Make sure you stop using pbr's autodoc feature before converting > > > them to the new PTI for docs. > > > > > > [snip] > > > > > That's unfortunate. What we really need is a migration path from the > 'pbr' way of doing things to something else. I see three possible > avenues at this point in time: > >1. Start using 'sphinx.ext.autosummary'. Apparently this can do similar > things to 'sphinx-apidoc' but it takes the form of an extension. > From my brief experiments, the output generated from this is > radically different and far less comprehensive than what 'sphinx- > apidoc' generates. However, it supports templating so we could > probably configure this somehow and add our own special directive > somewhere like 'openstackdocstheme' >2. Push for the 'sphinx.ext.apidoc' extension I proposed some time back > against upstream Sphinx [1]. This essentially does what the PBR > extension does but moves configuration into 'conf.py'. However, this > is currently held up as I can't adequately explain the differences > between this and 'sphinx.ext.autosummary' (there's definite overlap > but I don't understand 'autosummary' well enough to compare them). >3. Modify the upstream jobs that detect the pbr integration and have > them run 'sphinx-apidoc' before 'sphinx-build'. This is the least > technically appealing approach as it still leaves us unable to build > stuff locally and adds yet more "magic" to the gate, but it does let > us progress. > It's not mentioned here, but I discovered today that Cinder is using the sphinx.ext.autodoc module. Is there any issue with using this? __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Following the new PTI for document build, broken local builds
Excerpts from Stephen Finucane's message of 2018-03-23 17:25:42 +: > On Fri, 2018-03-23 at 12:23 -0400, Doug Hellmann wrote: > > Excerpts from Monty Taylor's message of 2018-03-23 08:03:22 -0500: > > > On 03/22/2018 05:43 AM, Stephen Finucane wrote: > > > > > > > > That's unfortunate. What we really need is a migration path from the > > > > 'pbr' way of doing things to something else. I see three possible > > > > avenues at this point in time: > > > > > > > > 1. Start using 'sphinx.ext.autosummary'. Apparently this can do > > > > similar > > > >things to 'sphinx-apidoc' but it takes the form of an extension. > > > >From my brief experiments, the output generated from this is > > > >radically different and far less comprehensive than what 'sphinx- > > > >apidoc' generates. However, it supports templating so we could > > > >probably configure this somehow and add our own special directive > > > >somewhere like 'openstackdocstheme' > > > > 2. Push for the 'sphinx.ext.apidoc' extension I proposed some time > > > > back > > > >against upstream Sphinx [1]. This essentially does what the PBR > > > >extension does but moves configuration into 'conf.py'. However, > > > > this > > > >is currently held up as I can't adequately explain the > > > > differences > > > >between this and 'sphinx.ext.autosummary' (there's definite > > > > overlap > > > >but I don't understand 'autosummary' well enough to compare > > > > them). > > > > 3. Modify the upstream jobs that detect the pbr integration and have > > > >them run 'sphinx-apidoc' before 'sphinx-build'. This is the least > > > >technically appealing approach as it still leaves us unable to > > > > build > > > >stuff locally and adds yet more "magic" to the gate, but it does > > > > let > > > >us progress. > > > > > > I'd suggest a #4: > > > > > > Take the sphinx.ext.apidoc extension and make it a standalone extension > > > people can add to doc/requirements.txt and conf.py. That way we don't > > > have to convince the sphinx folks to land it. > > > > > > I'd been thinking for a while "we should just write a sphinx extension > > > with the pbr logic in it" - but hadn't gotten around to doing anything > > > about it. If you've already written that extension - I think we're in > > > potentially great shape! > > > > That also has the benefit that we don't have to wait for a new sphinx > > release to start using it. > > I can do this. Where will it live? pbr? openstackdocstheme? Somewhere > else? > > Stephen > I think the idea is to make a new thing. If you put it in the sphinx-contrib org on github it will be easy for other people to contribute and use it. Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Following the new PTI for document build, broken local builds
On Fri, 2018-03-23 at 12:23 -0400, Doug Hellmann wrote: > Excerpts from Monty Taylor's message of 2018-03-23 08:03:22 -0500: > > On 03/22/2018 05:43 AM, Stephen Finucane wrote: > > > > > > That's unfortunate. What we really need is a migration path from the > > > 'pbr' way of doing things to something else. I see three possible > > > avenues at this point in time: > > > > > > 1. Start using 'sphinx.ext.autosummary'. Apparently this can do > > > similar > > >things to 'sphinx-apidoc' but it takes the form of an extension. > > >From my brief experiments, the output generated from this is > > >radically different and far less comprehensive than what 'sphinx- > > >apidoc' generates. However, it supports templating so we could > > >probably configure this somehow and add our own special directive > > >somewhere like 'openstackdocstheme' > > > 2. Push for the 'sphinx.ext.apidoc' extension I proposed some time > > > back > > >against upstream Sphinx [1]. This essentially does what the PBR > > >extension does but moves configuration into 'conf.py'. However, > > > this > > >is currently held up as I can't adequately explain the differences > > >between this and 'sphinx.ext.autosummary' (there's definite overlap > > >but I don't understand 'autosummary' well enough to compare them). > > > 3. Modify the upstream jobs that detect the pbr integration and have > > >them run 'sphinx-apidoc' before 'sphinx-build'. This is the least > > >technically appealing approach as it still leaves us unable to > > > build > > >stuff locally and adds yet more "magic" to the gate, but it does > > > let > > >us progress. > > > > I'd suggest a #4: > > > > Take the sphinx.ext.apidoc extension and make it a standalone extension > > people can add to doc/requirements.txt and conf.py. That way we don't > > have to convince the sphinx folks to land it. > > > > I'd been thinking for a while "we should just write a sphinx extension > > with the pbr logic in it" - but hadn't gotten around to doing anything > > about it. If you've already written that extension - I think we're in > > potentially great shape! > > That also has the benefit that we don't have to wait for a new sphinx > release to start using it. I can do this. Where will it live? pbr? openstackdocstheme? Somewhere else? Stephen __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Following the new PTI for document build, broken local builds
Excerpts from Monty Taylor's message of 2018-03-23 08:03:22 -0500: > On 03/22/2018 05:43 AM, Stephen Finucane wrote: > > > > That's unfortunate. What we really need is a migration path from the > > 'pbr' way of doing things to something else. I see three possible > > avenues at this point in time: > > > > 1. Start using 'sphinx.ext.autosummary'. Apparently this can do similar > >things to 'sphinx-apidoc' but it takes the form of an extension. > >From my brief experiments, the output generated from this is > >radically different and far less comprehensive than what 'sphinx- > >apidoc' generates. However, it supports templating so we could > >probably configure this somehow and add our own special directive > >somewhere like 'openstackdocstheme' > > 2. Push for the 'sphinx.ext.apidoc' extension I proposed some time back > >against upstream Sphinx [1]. This essentially does what the PBR > >extension does but moves configuration into 'conf.py'. However, this > >is currently held up as I can't adequately explain the differences > >between this and 'sphinx.ext.autosummary' (there's definite overlap > >but I don't understand 'autosummary' well enough to compare them). > > 3. Modify the upstream jobs that detect the pbr integration and have > >them run 'sphinx-apidoc' before 'sphinx-build'. This is the least > >technically appealing approach as it still leaves us unable to build > >stuff locally and adds yet more "magic" to the gate, but it does let > >us progress. > > I'd suggest a #4: > > Take the sphinx.ext.apidoc extension and make it a standalone extension > people can add to doc/requirements.txt and conf.py. That way we don't > have to convince the sphinx folks to land it. > > I'd been thinking for a while "we should just write a sphinx extension > with the pbr logic in it" - but hadn't gotten around to doing anything > about it. If you've already written that extension - I think we're in > potentially great shape! That also has the benefit that we don't have to wait for a new sphinx release to start using it. Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Following the new PTI for document build, broken local builds
On 03/22/2018 05:43 AM, Stephen Finucane wrote: On Wed, 2018-03-21 at 09:57 -0500, Sean McGinnis wrote: On Wed, Mar 21, 2018 at 10:49:02AM +, Stephen Finucane wrote: tl;dr: Make sure you stop using pbr's autodoc feature before converting them to the new PTI for docs. [snip] I've gone through and proposed a couple of reverts to fix projects we've already broken. However, going forward, there are two things people should do to prevent issues like this popping up. Unfortunately this will not work to just revert the changes. That may fix things locally, but they will not pass in gate by going back to the old way. Any cases of this will have to actually be updated to not use the unsupported pieces you point out. But the doc builds will still need to be done the way they are now, as that is what the PTI requires at this point. That's unfortunate. What we really need is a migration path from the 'pbr' way of doing things to something else. I see three possible avenues at this point in time: 1. Start using 'sphinx.ext.autosummary'. Apparently this can do similar things to 'sphinx-apidoc' but it takes the form of an extension. From my brief experiments, the output generated from this is radically different and far less comprehensive than what 'sphinx- apidoc' generates. However, it supports templating so we could probably configure this somehow and add our own special directive somewhere like 'openstackdocstheme' 2. Push for the 'sphinx.ext.apidoc' extension I proposed some time back against upstream Sphinx [1]. This essentially does what the PBR extension does but moves configuration into 'conf.py'. However, this is currently held up as I can't adequately explain the differences between this and 'sphinx.ext.autosummary' (there's definite overlap but I don't understand 'autosummary' well enough to compare them). 3. Modify the upstream jobs that detect the pbr integration and have them run 'sphinx-apidoc' before 'sphinx-build'. This is the least technically appealing approach as it still leaves us unable to build stuff locally and adds yet more "magic" to the gate, but it does let us progress. I'd suggest a #4: Take the sphinx.ext.apidoc extension and make it a standalone extension people can add to doc/requirements.txt and conf.py. That way we don't have to convince the sphinx folks to land it. I'd been thinking for a while "we should just write a sphinx extension with the pbr logic in it" - but hadn't gotten around to doing anything about it. If you've already written that extension - I think we're in potentially great shape! Try as I may, I don't really have the bandwidth to work on this for another few weeks so I'd appreciate help from anyone with sufficient Sphinx-fu to come up with a long-term solution to this issue. Cheers, Stephen [1] https://github.com/sphinx-doc/sphinx/pull/4101/files * Firstly, you should remove the '[build_sphinx]' and '[pbr]' sections from 'setup.cfg' in any patches that aim to convert a project to use the new PTI. This will ensure the gate catches any potential issues. * In addition, if your project uses the pbr autodoc feature, you should either (a) remove these docs from your documentation tree or (b) migrate to something else like the 'sphinx.ext.autosummary' extension [5]. I aim to post instructions on the latter shortly. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Following the new PTI for document build, broken local builds
On 03/22/2018 04:39 PM, Sean McGinnis wrote: That's unfortunate. What we really need is a migration path from the 'pbr' way of doing things to something else. I see three possible avenues at this point in time: 1. Start using 'sphinx.ext.autosummary'. Apparently this can do similar things to 'sphinx-apidoc' but it takes the form of an extension. From my brief experiments, the output generated from this is radically different and far less comprehensive than what 'sphinx- apidoc' generates. However, it supports templating so we could probably configure this somehow and add our own special directive somewhere like 'openstackdocstheme' 2. Push for the 'sphinx.ext.apidoc' extension I proposed some time back against upstream Sphinx [1]. This essentially does what the PBR extension does but moves configuration into 'conf.py'. However, this is currently held up as I can't adequately explain the differences between this and 'sphinx.ext.autosummary' (there's definite overlap but I don't understand 'autosummary' well enough to compare them). 3. Modify the upstream jobs that detect the pbr integration and have them run 'sphinx-apidoc' before 'sphinx-build'. This is the least technically appealing approach as it still leaves us unable to build stuff locally and adds yet more "magic" to the gate, but it does let us progress. Try as I may, I don't really have the bandwidth to work on this for another few weeks so I'd appreciate help from anyone with sufficient Sphinx-fu to come up with a long-term solution to this issue. Cheers, Stephen I think we could probably go with 1 until and if 2 becomes an option. It does change output quite a bit. I played around with 3, but I think we will have enough differences between projects as to _where_ specifically this generated content needs to be placed that it will make that approach a little more brittle. One other things that comes to mind - I think most service projects, if they are even using this, could probably just drop it. I've found the generated "API" documentation for service modules to be of very limited use. That would at least narrow things down to lib projects. So this would still be an issue for the oslo libs for sure. In that case, you do what that module API documentation in most cases. This is also an issue for clients. I would kindly ask people doing this work to stop proposing patches that just remove the API reference without any replacement. But personally, I would encourage service projects to get around this issue by just not doing it. It would appear that would take care of a large chunk of the current usage: http://codesearch.openstack.org/?q=autodoc_index_modules&i=nope&files=setup.cfg&repos= __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Following the new PTI for document build, broken local builds
> > > > That's unfortunate. What we really need is a migration path from the > > 'pbr' way of doing things to something else. I see three possible > > avenues at this point in time: > > > >1. Start using 'sphinx.ext.autosummary'. Apparently this can do similar > > things to 'sphinx-apidoc' but it takes the form of an extension. > > From my brief experiments, the output generated from this is > > radically different and far less comprehensive than what 'sphinx- > > apidoc' generates. However, it supports templating so we could > > probably configure this somehow and add our own special directive > > somewhere like 'openstackdocstheme' > >2. Push for the 'sphinx.ext.apidoc' extension I proposed some time back > > against upstream Sphinx [1]. This essentially does what the PBR > > extension does but moves configuration into 'conf.py'. However, this > > is currently held up as I can't adequately explain the differences > > between this and 'sphinx.ext.autosummary' (there's definite overlap > > but I don't understand 'autosummary' well enough to compare them). > >3. Modify the upstream jobs that detect the pbr integration and have > > them run 'sphinx-apidoc' before 'sphinx-build'. This is the least > > technically appealing approach as it still leaves us unable to build > > stuff locally and adds yet more "magic" to the gate, but it does let > > us progress. > > > > Try as I may, I don't really have the bandwidth to work on this for > > another few weeks so I'd appreciate help from anyone with sufficient > > Sphinx-fu to come up with a long-term solution to this issue. > > > > Cheers, > > Stephen > > > > I think we could probably go with 1 until and if 2 becomes an option. It does > change output quite a bit. > > I played around with 3, but I think we will have enough differences between > projects as to _where_ specifically this generated content needs to be placed > that it will make that approach a little more brittle. > One other things that comes to mind - I think most service projects, if they are even using this, could probably just drop it. I've found the generated "API" documentation for service modules to be of very limited use. That would at least narrow things down to lib projects. So this would still be an issue for the oslo libs for sure. In that case, you do what that module API documentation in most cases. But personally, I would encourage service projects to get around this issue by just not doing it. It would appear that would take care of a large chunk of the current usage: http://codesearch.openstack.org/?q=autodoc_index_modules&i=nope&files=setup.cfg&repos= __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Following the new PTI for document build, broken local builds
> > > > Unfortunately this will not work to just revert the changes. That may fix > > things locally, but they will not pass in gate by going back to the old way. > > > > Any cases of this will have to actually be updated to not use the > > unsupported > > pieces you point out. But the doc builds will still need to be done the way > > they are now, as that is what the PTI requires at this point. > > That's unfortunate. What we really need is a migration path from the > 'pbr' way of doing things to something else. I see three possible > avenues at this point in time: > >1. Start using 'sphinx.ext.autosummary'. Apparently this can do similar > things to 'sphinx-apidoc' but it takes the form of an extension. > From my brief experiments, the output generated from this is > radically different and far less comprehensive than what 'sphinx- > apidoc' generates. However, it supports templating so we could > probably configure this somehow and add our own special directive > somewhere like 'openstackdocstheme' >2. Push for the 'sphinx.ext.apidoc' extension I proposed some time back > against upstream Sphinx [1]. This essentially does what the PBR > extension does but moves configuration into 'conf.py'. However, this > is currently held up as I can't adequately explain the differences > between this and 'sphinx.ext.autosummary' (there's definite overlap > but I don't understand 'autosummary' well enough to compare them). >3. Modify the upstream jobs that detect the pbr integration and have > them run 'sphinx-apidoc' before 'sphinx-build'. This is the least > technically appealing approach as it still leaves us unable to build > stuff locally and adds yet more "magic" to the gate, but it does let > us progress. > > Try as I may, I don't really have the bandwidth to work on this for > another few weeks so I'd appreciate help from anyone with sufficient > Sphinx-fu to come up with a long-term solution to this issue. > > Cheers, > Stephen > I think we could probably go with 1 until and if 2 becomes an option. It does change output quite a bit. I played around with 3, but I think we will have enough differences between projects as to _where_ specifically this generated content needs to be placed that it will make that approach a little more brittle. All that said, I think once we decide on a path and get something out there, there does seem to be a group of folks that are more than willing to follow that established pattern and desseminate it out to all other projects. We just need to decide I guess. Your sphinx-fu is probably stronger than mine, so hopefully someone else with more experience can chime in here. ;) Sean __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Following the new PTI for document build, broken local builds
On Wed, 2018-03-21 at 09:57 -0500, Sean McGinnis wrote: > On Wed, Mar 21, 2018 at 10:49:02AM +, Stephen Finucane wrote: > > tl;dr: Make sure you stop using pbr's autodoc feature before converting > > them to the new PTI for docs. > > > > [snip] > > > > I've gone through and proposed a couple of reverts to fix projects > > we've already broken. However, going forward, there are two things > > people should do to prevent issues like this popping up. > > Unfortunately this will not work to just revert the changes. That may fix > things locally, but they will not pass in gate by going back to the old way. > > Any cases of this will have to actually be updated to not use the unsupported > pieces you point out. But the doc builds will still need to be done the way > they are now, as that is what the PTI requires at this point. That's unfortunate. What we really need is a migration path from the 'pbr' way of doing things to something else. I see three possible avenues at this point in time: 1. Start using 'sphinx.ext.autosummary'. Apparently this can do similar things to 'sphinx-apidoc' but it takes the form of an extension. From my brief experiments, the output generated from this is radically different and far less comprehensive than what 'sphinx- apidoc' generates. However, it supports templating so we could probably configure this somehow and add our own special directive somewhere like 'openstackdocstheme' 2. Push for the 'sphinx.ext.apidoc' extension I proposed some time back against upstream Sphinx [1]. This essentially does what the PBR extension does but moves configuration into 'conf.py'. However, this is currently held up as I can't adequately explain the differences between this and 'sphinx.ext.autosummary' (there's definite overlap but I don't understand 'autosummary' well enough to compare them). 3. Modify the upstream jobs that detect the pbr integration and have them run 'sphinx-apidoc' before 'sphinx-build'. This is the least technically appealing approach as it still leaves us unable to build stuff locally and adds yet more "magic" to the gate, but it does let us progress. Try as I may, I don't really have the bandwidth to work on this for another few weeks so I'd appreciate help from anyone with sufficient Sphinx-fu to come up with a long-term solution to this issue. Cheers, Stephen [1] https://github.com/sphinx-doc/sphinx/pull/4101/files > > * Firstly, you should remove the '[build_sphinx]' and '[pbr]' sections > >from 'setup.cfg' in any patches that aim to convert a project to use > >the new PTI. This will ensure the gate catches any potential > >issues. > > * In addition, if your project uses the pbr autodoc feature, you > >should either (a) remove these docs from your documentation tree or > >(b) migrate to something else like the 'sphinx.ext.autosummary' > >extension [5]. I aim to post instructions on the latter shortly. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Following the new PTI for document build, broken local builds
On Wed, Mar 21, 2018 at 10:49:02AM +, Stephen Finucane wrote: > tl;dr: Make sure you stop using pbr's autodoc feature before converting > them to the new PTI for docs. > > [snip] > > I've gone through and proposed a couple of reverts to fix projects > we've already broken. However, going forward, there are two things > people should do to prevent issues like this popping up. > Unfortunately this will not work to just revert the changes. That may fix things locally, but they will not pass in gate by going back to the old way. Any cases of this will have to actually be updated to not use the unsupported pieces you point out. But the doc builds will still need to be done the way they are now, as that is what the PTI requires at this point. > * Firstly, you should remove the '[build_sphinx]' and '[pbr]' sections >from 'setup.cfg' in any patches that aim to convert a project to use >the new PTI. This will ensure the gate catches any potential >issues. > * In addition, if your project uses the pbr autodoc feature, you >should either (a) remove these docs from your documentation tree or >(b) migrate to something else like the 'sphinx.ext.autosummary' >extension [5]. I aim to post instructions on the latter shortly. > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev