Re: [openstack-dev] [tc][docs][release] Updating the PTI for docs and tarballs
Excerpts from Monty Taylor's message of 2017-10-05 08:45:52 -0500: > On 09/30/2017 10:40 AM, Doug Hellmann wrote: > > Excerpts from Monty Taylor's message of 2017-09-30 10:20:08 -0500: > >> Hey everybody, > >> > >> Oh goodie, I can hear you say, let's definitely spend some time > >> bikeshedding about specific command invocations related to building docs > >> and tarballs!!! > >> > >> tl;dr I want to change the PTI for docs and tarball building to be less > >> OpenStack-specific > >> > >> The Problem > >> === > >> > >> As we work on Zuul v3, there are a set of job definitions that are > >> rather fundamental that can totally be directly shared between Zuul > >> installations whether those Zuuls are working with OpenStack content or > >> not. As an example "tox -epy27" is a fairly standard thing, so a Zuul > >> job called "tox-py27" has no qualities specific to OpenStack and could > >> realistically be used by anyone who uses tox to manage their project. > >> > >> Docs and Tarballs builds for us, however, are the following: > >> > >> tox -evenv -- python setup.py sdist > >> tox -evenv -- python setup.py build_sphinx > >> > >> Neither of those are things that are likely to work outside of > >> OpenStack. (The 'venv' tox environment is not a default tox thing) > >> > >> I'm going to argue neither of them are actually providing us with much > >> value. > >> > >> Tarball Creation > >> > >> > >> Tarball creation is super simple. setup_requires is already handled out > >> of band of everything else. Go clone nova into a completely empty system > >> and run python setup.py sdist ... and it works. (actually, nova is big. > >> use something smaller like gertty ...) > >> > >> docker run -it --rm python bash -c 'git clone \ > >>https://git.openstack.org/openstack/gertty && cd gertty \ > >>&& python setup.py sdist' > >> > >> There is not much value in that tox wrapper - and it's not like it's > >> making it EASIER to run the command. In fact, it's more typing. > >> > >> I propose we change the PTI from: > >> > >> tox -evenv python setup.py sdist > >> > >> to: > >> > >> python setup.py sdist > >> > >> and then change the gate jobs to use the non-tox form of the command. > >> > >> I'd also like to further change it to be explicit that we also build > >> wheels. So the ACTUAL commands that the project should support are: > >> > >> python setup.py sdist > >> python setup.py bdist_wheel > >> > >> All of our projects support this already, so this should be a no-op. > >> > >> Notes: > >> > >> * Python projects that need to build C extensions might need their pip > >> requirements (and bindep requirements) installed in order to run > >> bdist_wheel. We do not support that broadly at the moment ANYWAY - so > >> I'd like to leave that as an outlier and handle it when we need to > >> handle it. > >> > >> * It's *possible* that somewhere we have a repo that has somehow done > >> something that would cause python setup.py sdist or python setup.py > >> bdist_wheel to not work without pip requirements installed. I believe we > >> should consider that a bug and fix it in the project if we find such a > >> thing - but since we use pbr in all of the OpenStack projects, I find it > >> extremely unlikely. > >> > >> Governance patch submitted: https://review.openstack.org/508693 > >> > >> Sphinx Documentation > >> > >> > >> Doc builds are more complex - but I think there is a high amount of > >> value in changing how we invoke them for a few reasons. > >> > >> a) nobody uses 'tox -evenv -- python setup.py build_sphinx' but us > >> b) we decided to use sphinx for go and javascript - but we invoke sphinx > >> differently for each of those (since they naturally don't have tox), > >> meaning we can't just have a "build-sphinx-docs" job and even share it > >> with ourselves. > >> c) readthedocs.org is an excellent Open Source site that builds and > >> hosts sphinx docs for projects. They have an interface for docs > >> requirements documented and defined that we can align. By aligning, > >> projects can use migrate between docs.o.o and readthedocs.org and still > >> have a consistent experience. > >> > >> The PTI I'd like to propose for this is more complex, so I'd like to > >> describe it in terms of: > >> > >> - OpenStack organizational requirements > >> - helper sugar for developers with per-language recommendations > >> > >> I believe the result can be a single in-tree doc PTI that applies to > >> python, go and javascript - and a single Zuul job that applies to all of > >> our projects AND non-OpenStack projects as well. > >> > >> Organizational Requirements > >> --- > >> > >> Following readthedocs.org logic we can actually support a wider range of > >> schemes technically, but there is human value in having consistency on > >> these topics across our OpenStack repos. > >> > >> * docs live in doc/source > >> * Python requirements
Re: [openstack-dev] [tc][docs][release] Updating the PTI for docs and tarballs
On 09/30/2017 10:40 AM, Doug Hellmann wrote: Excerpts from Monty Taylor's message of 2017-09-30 10:20:08 -0500: Hey everybody, Oh goodie, I can hear you say, let's definitely spend some time bikeshedding about specific command invocations related to building docs and tarballs!!! tl;dr I want to change the PTI for docs and tarball building to be less OpenStack-specific The Problem === As we work on Zuul v3, there are a set of job definitions that are rather fundamental that can totally be directly shared between Zuul installations whether those Zuuls are working with OpenStack content or not. As an example "tox -epy27" is a fairly standard thing, so a Zuul job called "tox-py27" has no qualities specific to OpenStack and could realistically be used by anyone who uses tox to manage their project. Docs and Tarballs builds for us, however, are the following: tox -evenv -- python setup.py sdist tox -evenv -- python setup.py build_sphinx Neither of those are things that are likely to work outside of OpenStack. (The 'venv' tox environment is not a default tox thing) I'm going to argue neither of them are actually providing us with much value. Tarball Creation Tarball creation is super simple. setup_requires is already handled out of band of everything else. Go clone nova into a completely empty system and run python setup.py sdist ... and it works. (actually, nova is big. use something smaller like gertty ...) docker run -it --rm python bash -c 'git clone \ https://git.openstack.org/openstack/gertty && cd gertty \ && python setup.py sdist' There is not much value in that tox wrapper - and it's not like it's making it EASIER to run the command. In fact, it's more typing. I propose we change the PTI from: tox -evenv python setup.py sdist to: python setup.py sdist and then change the gate jobs to use the non-tox form of the command. I'd also like to further change it to be explicit that we also build wheels. So the ACTUAL commands that the project should support are: python setup.py sdist python setup.py bdist_wheel All of our projects support this already, so this should be a no-op. Notes: * Python projects that need to build C extensions might need their pip requirements (and bindep requirements) installed in order to run bdist_wheel. We do not support that broadly at the moment ANYWAY - so I'd like to leave that as an outlier and handle it when we need to handle it. * It's *possible* that somewhere we have a repo that has somehow done something that would cause python setup.py sdist or python setup.py bdist_wheel to not work without pip requirements installed. I believe we should consider that a bug and fix it in the project if we find such a thing - but since we use pbr in all of the OpenStack projects, I find it extremely unlikely. Governance patch submitted: https://review.openstack.org/508693 Sphinx Documentation Doc builds are more complex - but I think there is a high amount of value in changing how we invoke them for a few reasons. a) nobody uses 'tox -evenv -- python setup.py build_sphinx' but us b) we decided to use sphinx for go and javascript - but we invoke sphinx differently for each of those (since they naturally don't have tox), meaning we can't just have a "build-sphinx-docs" job and even share it with ourselves. c) readthedocs.org is an excellent Open Source site that builds and hosts sphinx docs for projects. They have an interface for docs requirements documented and defined that we can align. By aligning, projects can use migrate between docs.o.o and readthedocs.org and still have a consistent experience. The PTI I'd like to propose for this is more complex, so I'd like to describe it in terms of: - OpenStack organizational requirements - helper sugar for developers with per-language recommendations I believe the result can be a single in-tree doc PTI that applies to python, go and javascript - and a single Zuul job that applies to all of our projects AND non-OpenStack projects as well. Organizational Requirements --- Following readthedocs.org logic we can actually support a wider range of schemes technically, but there is human value in having consistency on these topics across our OpenStack repos. * docs live in doc/source * Python requirements needed by Sphinx to build the docs live in doc/requirements.txt If the project is python: * doc/requirements.txt can assume the project will have been installed * The following should be set in setup.cfg: [build_sphinx] source-dir = doc/source build-dir = doc/build Doing the above allows the following commands to work cleanly in automation no matter what the language is: [ -f doc/requirements.txt ] && pip install -rdoc/requirements.txt sphinx-build -b doc/source doc/build I suspect you mean "sphinx-build -b html doc/source doc/build" (the "html" arg was missing). Yes! Please amend all
Re: [openstack-dev] [tc][docs][release] Updating the PTI for docs and tarballs
Having a PDF (or similar offline copy) was also requested during OpenStack UK days event during the executive Q with jbryce. Tim -Original Message- From: Doug HellmannReply-To: "OpenStack Development Mailing List (not for usage questions)" Date: Saturday, 30 September 2017 at 17:44 To: openstack-dev Subject: Re: [openstack-dev] [tc][docs][release] Updating the PTI for docs and tarballs Excerpts from Monty Taylor's message of 2017-09-30 10:20:08 -0500: > Hey everybody, > > Oh goodie, I can hear you say, let's definitely spend some time > bikeshedding about specific command invocations related to building docs > and tarballs!!! > > tl;dr I want to change the PTI for docs and tarball building to be less > OpenStack-specific > > The Problem > === > > As we work on Zuul v3, there are a set of job definitions that are > rather fundamental that can totally be directly shared between Zuul > installations whether those Zuuls are working with OpenStack content or > not. As an example "tox -epy27" is a fairly standard thing, so a Zuul > job called "tox-py27" has no qualities specific to OpenStack and could > realistically be used by anyone who uses tox to manage their project. > > Docs and Tarballs builds for us, however, are the following: > > tox -evenv -- python setup.py sdist > tox -evenv -- python setup.py build_sphinx > > Neither of those are things that are likely to work outside of > OpenStack. (The 'venv' tox environment is not a default tox thing) > > I'm going to argue neither of them are actually providing us with much > value. > > Tarball Creation > > > Tarball creation is super simple. setup_requires is already handled out > of band of everything else. Go clone nova into a completely empty system > and run python setup.py sdist ... and it works. (actually, nova is big. > use something smaller like gertty ...) > > docker run -it --rm python bash -c 'git clone \ > https://git.openstack.org/openstack/gertty && cd gertty \ > && python setup.py sdist' > > There is not much value in that tox wrapper - and it's not like it's > making it EASIER to run the command. In fact, it's more typing. > > I propose we change the PTI from: > >tox -evenv python setup.py sdist > > to: > >python setup.py sdist > > and then change the gate jobs to use the non-tox form of the command. > > I'd also like to further change it to be explicit that we also build > wheels. So the ACTUAL commands that the project should support are: > >python setup.py sdist >python setup.py bdist_wheel > > All of our projects support this already, so this should be a no-op. > > Notes: > > * Python projects that need to build C extensions might need their pip > requirements (and bindep requirements) installed in order to run > bdist_wheel. We do not support that broadly at the moment ANYWAY - so > I'd like to leave that as an outlier and handle it when we need to > handle it. > > * It's *possible* that somewhere we have a repo that has somehow done > something that would cause python setup.py sdist or python setup.py > bdist_wheel to not work without pip requirements installed. I believe we > should consider that a bug and fix it in the project if we find such a > thing - but since we use pbr in all of the OpenStack projects, I find it > extremely unlikely. > > Governance patch submitted: https://review.openstack.org/508693 > > Sphinx Documentation > > > Doc builds are more complex - but I think there is a high amount of > value in changing how we invoke them for a few reasons. > > a) nobody uses 'tox -evenv -- python setup.py build_sphinx' but us > b) we decided to use sphinx for go and javascript - but we invoke sphinx > differently for each of those (since they naturally don't have tox), > meaning we can't just have a "build-sphinx-docs" job and even share it > with ourselves. > c) readthedocs.org is an excellent Open Source site that builds and > hosts sphinx docs for projects. They have an interface for docs > requirements documented and defined that we can align. By aligning, > projects can use migrate between docs.o.o and readthedocs.org and still > have a consistent experience. > > The PTI I'd like to propose for this is more complex, so I'd like to > describe it in terms of: > > - OpenStack organizational requirements > - helper sugar for developers with per-language recommendations >
Re: [openstack-dev] [tc][docs][release] Updating the PTI for docs and tarballs
Excerpts from Monty Taylor's message of 2017-09-30 10:20:08 -0500: > Hey everybody, > > Oh goodie, I can hear you say, let's definitely spend some time > bikeshedding about specific command invocations related to building docs > and tarballs!!! > > tl;dr I want to change the PTI for docs and tarball building to be less > OpenStack-specific > > The Problem > === > > As we work on Zuul v3, there are a set of job definitions that are > rather fundamental that can totally be directly shared between Zuul > installations whether those Zuuls are working with OpenStack content or > not. As an example "tox -epy27" is a fairly standard thing, so a Zuul > job called "tox-py27" has no qualities specific to OpenStack and could > realistically be used by anyone who uses tox to manage their project. > > Docs and Tarballs builds for us, however, are the following: > > tox -evenv -- python setup.py sdist > tox -evenv -- python setup.py build_sphinx > > Neither of those are things that are likely to work outside of > OpenStack. (The 'venv' tox environment is not a default tox thing) > > I'm going to argue neither of them are actually providing us with much > value. > > Tarball Creation > > > Tarball creation is super simple. setup_requires is already handled out > of band of everything else. Go clone nova into a completely empty system > and run python setup.py sdist ... and it works. (actually, nova is big. > use something smaller like gertty ...) > > docker run -it --rm python bash -c 'git clone \ > https://git.openstack.org/openstack/gertty && cd gertty \ > && python setup.py sdist' > > There is not much value in that tox wrapper - and it's not like it's > making it EASIER to run the command. In fact, it's more typing. > > I propose we change the PTI from: > >tox -evenv python setup.py sdist > > to: > >python setup.py sdist > > and then change the gate jobs to use the non-tox form of the command. > > I'd also like to further change it to be explicit that we also build > wheels. So the ACTUAL commands that the project should support are: > >python setup.py sdist >python setup.py bdist_wheel > > All of our projects support this already, so this should be a no-op. > > Notes: > > * Python projects that need to build C extensions might need their pip > requirements (and bindep requirements) installed in order to run > bdist_wheel. We do not support that broadly at the moment ANYWAY - so > I'd like to leave that as an outlier and handle it when we need to > handle it. > > * It's *possible* that somewhere we have a repo that has somehow done > something that would cause python setup.py sdist or python setup.py > bdist_wheel to not work without pip requirements installed. I believe we > should consider that a bug and fix it in the project if we find such a > thing - but since we use pbr in all of the OpenStack projects, I find it > extremely unlikely. > > Governance patch submitted: https://review.openstack.org/508693 > > Sphinx Documentation > > > Doc builds are more complex - but I think there is a high amount of > value in changing how we invoke them for a few reasons. > > a) nobody uses 'tox -evenv -- python setup.py build_sphinx' but us > b) we decided to use sphinx for go and javascript - but we invoke sphinx > differently for each of those (since they naturally don't have tox), > meaning we can't just have a "build-sphinx-docs" job and even share it > with ourselves. > c) readthedocs.org is an excellent Open Source site that builds and > hosts sphinx docs for projects. They have an interface for docs > requirements documented and defined that we can align. By aligning, > projects can use migrate between docs.o.o and readthedocs.org and still > have a consistent experience. > > The PTI I'd like to propose for this is more complex, so I'd like to > describe it in terms of: > > - OpenStack organizational requirements > - helper sugar for developers with per-language recommendations > > I believe the result can be a single in-tree doc PTI that applies to > python, go and javascript - and a single Zuul job that applies to all of > our projects AND non-OpenStack projects as well. > > Organizational Requirements > --- > > Following readthedocs.org logic we can actually support a wider range of > schemes technically, but there is human value in having consistency on > these topics across our OpenStack repos. > > * docs live in doc/source > * Python requirements needed by Sphinx to build the docs live in > doc/requirements.txt > > If the project is python: > > * doc/requirements.txt can assume the project will have been installed > * The following should be set in setup.cfg: > >[build_sphinx] >source-dir = doc/source >build-dir = doc/build > > Doing the above allows the following commands to work cleanly in > automation no matter what the
Re: [openstack-dev] [tc][docs][release] Updating the PTI for docs and tarballs
On 2017-09-30 10:20:08 -0500 (-0500), Monty Taylor wrote: [...] > I want to change the PTI for docs and tarball building to be less > OpenStack-specific [...] Looks great! Anything to make OpenStack less snowflake-like and our build processes more familiar to developers from other communities. -- Jeremy Stanley signature.asc Description: Digital signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [tc][docs][release] Updating the PTI for docs and tarballs
Hey everybody, Oh goodie, I can hear you say, let's definitely spend some time bikeshedding about specific command invocations related to building docs and tarballs!!! tl;dr I want to change the PTI for docs and tarball building to be less OpenStack-specific The Problem === As we work on Zuul v3, there are a set of job definitions that are rather fundamental that can totally be directly shared between Zuul installations whether those Zuuls are working with OpenStack content or not. As an example "tox -epy27" is a fairly standard thing, so a Zuul job called "tox-py27" has no qualities specific to OpenStack and could realistically be used by anyone who uses tox to manage their project. Docs and Tarballs builds for us, however, are the following: tox -evenv -- python setup.py sdist tox -evenv -- python setup.py build_sphinx Neither of those are things that are likely to work outside of OpenStack. (The 'venv' tox environment is not a default tox thing) I'm going to argue neither of them are actually providing us with much value. Tarball Creation Tarball creation is super simple. setup_requires is already handled out of band of everything else. Go clone nova into a completely empty system and run python setup.py sdist ... and it works. (actually, nova is big. use something smaller like gertty ...) docker run -it --rm python bash -c 'git clone \ https://git.openstack.org/openstack/gertty && cd gertty \ && python setup.py sdist' There is not much value in that tox wrapper - and it's not like it's making it EASIER to run the command. In fact, it's more typing. I propose we change the PTI from: tox -evenv python setup.py sdist to: python setup.py sdist and then change the gate jobs to use the non-tox form of the command. I'd also like to further change it to be explicit that we also build wheels. So the ACTUAL commands that the project should support are: python setup.py sdist python setup.py bdist_wheel All of our projects support this already, so this should be a no-op. Notes: * Python projects that need to build C extensions might need their pip requirements (and bindep requirements) installed in order to run bdist_wheel. We do not support that broadly at the moment ANYWAY - so I'd like to leave that as an outlier and handle it when we need to handle it. * It's *possible* that somewhere we have a repo that has somehow done something that would cause python setup.py sdist or python setup.py bdist_wheel to not work without pip requirements installed. I believe we should consider that a bug and fix it in the project if we find such a thing - but since we use pbr in all of the OpenStack projects, I find it extremely unlikely. Governance patch submitted: https://review.openstack.org/508693 Sphinx Documentation Doc builds are more complex - but I think there is a high amount of value in changing how we invoke them for a few reasons. a) nobody uses 'tox -evenv -- python setup.py build_sphinx' but us b) we decided to use sphinx for go and javascript - but we invoke sphinx differently for each of those (since they naturally don't have tox), meaning we can't just have a "build-sphinx-docs" job and even share it with ourselves. c) readthedocs.org is an excellent Open Source site that builds and hosts sphinx docs for projects. They have an interface for docs requirements documented and defined that we can align. By aligning, projects can use migrate between docs.o.o and readthedocs.org and still have a consistent experience. The PTI I'd like to propose for this is more complex, so I'd like to describe it in terms of: - OpenStack organizational requirements - helper sugar for developers with per-language recommendations I believe the result can be a single in-tree doc PTI that applies to python, go and javascript - and a single Zuul job that applies to all of our projects AND non-OpenStack projects as well. Organizational Requirements --- Following readthedocs.org logic we can actually support a wider range of schemes technically, but there is human value in having consistency on these topics across our OpenStack repos. * docs live in doc/source * Python requirements needed by Sphinx to build the docs live in doc/requirements.txt If the project is python: * doc/requirements.txt can assume the project will have been installed * The following should be set in setup.cfg: [build_sphinx] source-dir = doc/source build-dir = doc/build Doing the above allows the following commands to work cleanly in automation no matter what the language is: [ -f doc/requirements.txt ] && pip install -rdoc/requirements.txt sphinx-build -b doc/source doc/build No additional commands should be required. The setup.cfg stanza allows: python setup.py build_sphinx to continue to work. (also, everyone already has one) Helper sugar for developers