Re: [openstack-dev] [tc][docs][release] Updating the PTI for docs and tarballs

2017-10-09 Thread Doug Hellmann
Excerpts from Monty Taylor's message of 2017-10-05 08:45:52 -0500:
> On 09/30/2017 10:40 AM, Doug Hellmann wrote:
> > Excerpts from Monty Taylor's message of 2017-09-30 10:20:08 -0500:
> >> Hey everybody,
> >>
> >> Oh goodie, I can hear you say, let's definitely spend some time
> >> bikeshedding about specific command invocations related to building docs
> >> and tarballs!!!
> >>
> >> tl;dr I want to change the PTI for docs and tarball building to be less
> >> OpenStack-specific
> >>
> >> The Problem
> >> ===
> >>
> >> As we work on Zuul v3, there are a set of job definitions that are
> >> rather fundamental that can totally be directly shared between Zuul
> >> installations whether those Zuuls are working with OpenStack content or
> >> not. As an example "tox -epy27" is a fairly standard thing, so a Zuul
> >> job called "tox-py27" has no qualities specific to OpenStack and could
> >> realistically be used by anyone who uses tox to manage their project.
> >>
> >> Docs and Tarballs builds for us, however, are the following:
> >>
> >> tox -evenv -- python setup.py sdist
> >> tox -evenv -- python setup.py build_sphinx
> >>
> >> Neither of those are things that are likely to work outside of
> >> OpenStack. (The 'venv' tox environment is not a default tox thing)
> >>
> >> I'm going to argue neither of them are actually providing us with much
> >> value.
> >>
> >> Tarball Creation
> >> 
> >>
> >> Tarball creation is super simple. setup_requires is already handled out
> >> of band of everything else. Go clone nova into a completely empty system
> >> and run python setup.py sdist ... and it works. (actually, nova is big.
> >> use something smaller like gertty ...)
> >>
> >>  docker run -it --rm python bash -c 'git clone \
> >>https://git.openstack.org/openstack/gertty && cd gertty \
> >>&& python setup.py sdist'
> >>
> >> There is not much value in that tox wrapper - and it's not like it's
> >> making it EASIER to run the command. In fact, it's more typing.
> >>
> >> I propose we change the PTI from:
> >>
> >> tox -evenv python setup.py sdist
> >>
> >> to:
> >>
> >> python setup.py sdist
> >>
> >> and then change the gate jobs to use the non-tox form of the command.
> >>
> >> I'd also like to further change it to be explicit that we also build
> >> wheels. So the ACTUAL commands that the project should support are:
> >>
> >> python setup.py sdist
> >> python setup.py bdist_wheel
> >>
> >> All of our projects support this already, so this should be a no-op.
> >>
> >> Notes:
> >>
> >> * Python projects that need to build C extensions might need their pip
> >> requirements (and bindep requirements) installed in order to run
> >> bdist_wheel. We do not support that broadly at the moment ANYWAY - so
> >> I'd like to leave that as an outlier and handle it when we need to
> >> handle it.
> >>
> >> * It's *possible* that somewhere we have a repo that has somehow done
> >> something that would cause python setup.py sdist or python setup.py
> >> bdist_wheel to not work without pip requirements installed. I believe we
> >> should consider that a bug and fix it in the project if we find such a
> >> thing - but since we use pbr in all of the OpenStack projects, I find it
> >> extremely unlikely.
> >>
> >> Governance patch submitted: https://review.openstack.org/508693
> >>
> >> Sphinx Documentation
> >> 
> >>
> >> Doc builds are more complex - but I think there is a high amount of
> >> value in changing how we invoke them for a few reasons.
> >>
> >> a) nobody uses 'tox -evenv -- python setup.py build_sphinx' but us
> >> b) we decided to use sphinx for go and javascript - but we invoke sphinx
> >> differently for each of those (since they naturally don't have tox),
> >> meaning we can't just have a "build-sphinx-docs" job and even share it
> >> with ourselves.
> >> c) readthedocs.org is an excellent Open Source site that builds and
> >> hosts sphinx docs for projects. They have an interface for docs
> >> requirements documented and defined that we can align. By aligning,
> >> projects can use migrate between docs.o.o and readthedocs.org and still
> >> have a consistent experience.
> >>
> >> The PTI I'd like to propose for this is more complex, so I'd like to
> >> describe it in terms of:
> >>
> >> - OpenStack organizational requirements
> >> - helper sugar for developers with per-language recommendations
> >>
> >> I believe the result can be a single in-tree doc PTI that applies to
> >> python, go and javascript - and a single Zuul job that applies to all of
> >> our projects AND non-OpenStack projects as well.
> >>
> >> Organizational Requirements
> >> ---
> >>
> >> Following readthedocs.org logic we can actually support a wider range of
> >> schemes technically, but there is human value in having consistency on
> >> these topics across our OpenStack repos.
> >>
> >> * docs live in doc/source
> >> * Python requirements 

Re: [openstack-dev] [tc][docs][release] Updating the PTI for docs and tarballs

2017-10-05 Thread Monty Taylor

On 09/30/2017 10:40 AM, Doug Hellmann wrote:

Excerpts from Monty Taylor's message of 2017-09-30 10:20:08 -0500:

Hey everybody,

Oh goodie, I can hear you say, let's definitely spend some time
bikeshedding about specific command invocations related to building docs
and tarballs!!!

tl;dr I want to change the PTI for docs and tarball building to be less
OpenStack-specific

The Problem
===

As we work on Zuul v3, there are a set of job definitions that are
rather fundamental that can totally be directly shared between Zuul
installations whether those Zuuls are working with OpenStack content or
not. As an example "tox -epy27" is a fairly standard thing, so a Zuul
job called "tox-py27" has no qualities specific to OpenStack and could
realistically be used by anyone who uses tox to manage their project.

Docs and Tarballs builds for us, however, are the following:

tox -evenv -- python setup.py sdist
tox -evenv -- python setup.py build_sphinx

Neither of those are things that are likely to work outside of
OpenStack. (The 'venv' tox environment is not a default tox thing)

I'm going to argue neither of them are actually providing us with much
value.

Tarball Creation


Tarball creation is super simple. setup_requires is already handled out
of band of everything else. Go clone nova into a completely empty system
and run python setup.py sdist ... and it works. (actually, nova is big.
use something smaller like gertty ...)

 docker run -it --rm python bash -c 'git clone \
   https://git.openstack.org/openstack/gertty && cd gertty \
   && python setup.py sdist'

There is not much value in that tox wrapper - and it's not like it's
making it EASIER to run the command. In fact, it's more typing.

I propose we change the PTI from:

tox -evenv python setup.py sdist

to:

python setup.py sdist

and then change the gate jobs to use the non-tox form of the command.

I'd also like to further change it to be explicit that we also build
wheels. So the ACTUAL commands that the project should support are:

python setup.py sdist
python setup.py bdist_wheel

All of our projects support this already, so this should be a no-op.

Notes:

* Python projects that need to build C extensions might need their pip
requirements (and bindep requirements) installed in order to run
bdist_wheel. We do not support that broadly at the moment ANYWAY - so
I'd like to leave that as an outlier and handle it when we need to
handle it.

* It's *possible* that somewhere we have a repo that has somehow done
something that would cause python setup.py sdist or python setup.py
bdist_wheel to not work without pip requirements installed. I believe we
should consider that a bug and fix it in the project if we find such a
thing - but since we use pbr in all of the OpenStack projects, I find it
extremely unlikely.

Governance patch submitted: https://review.openstack.org/508693

Sphinx Documentation


Doc builds are more complex - but I think there is a high amount of
value in changing how we invoke them for a few reasons.

a) nobody uses 'tox -evenv -- python setup.py build_sphinx' but us
b) we decided to use sphinx for go and javascript - but we invoke sphinx
differently for each of those (since they naturally don't have tox),
meaning we can't just have a "build-sphinx-docs" job and even share it
with ourselves.
c) readthedocs.org is an excellent Open Source site that builds and
hosts sphinx docs for projects. They have an interface for docs
requirements documented and defined that we can align. By aligning,
projects can use migrate between docs.o.o and readthedocs.org and still
have a consistent experience.

The PTI I'd like to propose for this is more complex, so I'd like to
describe it in terms of:

- OpenStack organizational requirements
- helper sugar for developers with per-language recommendations

I believe the result can be a single in-tree doc PTI that applies to
python, go and javascript - and a single Zuul job that applies to all of
our projects AND non-OpenStack projects as well.

Organizational Requirements
---

Following readthedocs.org logic we can actually support a wider range of
schemes technically, but there is human value in having consistency on
these topics across our OpenStack repos.

* docs live in doc/source
* Python requirements needed by Sphinx to build the docs live in
doc/requirements.txt

If the project is python:

* doc/requirements.txt can assume the project will have been installed
* The following should be set in setup.cfg:

[build_sphinx]
source-dir = doc/source
build-dir = doc/build

Doing the above allows the following commands to work cleanly in
automation no matter what the language is:

[ -f doc/requirements.txt ] && pip install -rdoc/requirements.txt
sphinx-build -b doc/source doc/build


I suspect you mean "sphinx-build -b html doc/source doc/build" (the
"html" arg was missing).


Yes! Please amend all 

Re: [openstack-dev] [tc][docs][release] Updating the PTI for docs and tarballs

2017-09-30 Thread Tim Bell
Having a PDF (or similar offline copy) was also requested during OpenStack UK 
days event during the executive Q with jbryce.

Tim

-Original Message-
From: Doug Hellmann 
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 

Date: Saturday, 30 September 2017 at 17:44
To: openstack-dev 
Subject: Re: [openstack-dev] [tc][docs][release] Updating the PTI for docs  
and tarballs

Excerpts from Monty Taylor's message of 2017-09-30 10:20:08 -0500:
> Hey everybody,
> 
> Oh goodie, I can hear you say, let's definitely spend some time 
> bikeshedding about specific command invocations related to building docs 
> and tarballs!!!
> 
> tl;dr I want to change the PTI for docs and tarball building to be less 
> OpenStack-specific
> 
> The Problem
> ===
> 
> As we work on Zuul v3, there are a set of job definitions that are 
> rather fundamental that can totally be directly shared between Zuul 
> installations whether those Zuuls are working with OpenStack content or 
> not. As an example "tox -epy27" is a fairly standard thing, so a Zuul 
> job called "tox-py27" has no qualities specific to OpenStack and could 
> realistically be used by anyone who uses tox to manage their project.
> 
> Docs and Tarballs builds for us, however, are the following:
> 
> tox -evenv -- python setup.py sdist
> tox -evenv -- python setup.py build_sphinx
> 
> Neither of those are things that are likely to work outside of 
> OpenStack. (The 'venv' tox environment is not a default tox thing)
> 
> I'm going to argue neither of them are actually providing us with much 
> value.
> 
> Tarball Creation
> 
> 
> Tarball creation is super simple. setup_requires is already handled out 
> of band of everything else. Go clone nova into a completely empty system 
> and run python setup.py sdist ... and it works. (actually, nova is big. 
> use something smaller like gertty ...)
> 
> docker run -it --rm python bash -c 'git clone \
>   https://git.openstack.org/openstack/gertty && cd gertty \
>   && python setup.py sdist'
> 
> There is not much value in that tox wrapper - and it's not like it's 
> making it EASIER to run the command. In fact, it's more typing.
> 
> I propose we change the PTI from:
> 
>tox -evenv python setup.py sdist
> 
> to:
> 
>python setup.py sdist
> 
> and then change the gate jobs to use the non-tox form of the command.
> 
> I'd also like to further change it to be explicit that we also build 
> wheels. So the ACTUAL commands that the project should support are:
> 
>python setup.py sdist
>python setup.py bdist_wheel
> 
> All of our projects support this already, so this should be a no-op.
> 
> Notes:
> 
> * Python projects that need to build C extensions might need their pip 
> requirements (and bindep requirements) installed in order to run 
> bdist_wheel. We do not support that broadly at the moment ANYWAY - so 
> I'd like to leave that as an outlier and handle it when we need to 
> handle it.
> 
> * It's *possible* that somewhere we have a repo that has somehow done 
> something that would cause python setup.py sdist or python setup.py 
> bdist_wheel to not work without pip requirements installed. I believe we 
> should consider that a bug and fix it in the project if we find such a 
> thing - but since we use pbr in all of the OpenStack projects, I find it 
> extremely unlikely.
> 
> Governance patch submitted: https://review.openstack.org/508693
> 
> Sphinx Documentation
> 
> 
> Doc builds are more complex - but I think there is a high amount of 
> value in changing how we invoke them for a few reasons.
> 
> a) nobody uses 'tox -evenv -- python setup.py build_sphinx' but us
> b) we decided to use sphinx for go and javascript - but we invoke sphinx 
> differently for each of those (since they naturally don't have tox), 
> meaning we can't just have a "build-sphinx-docs" job and even share it 
> with ourselves.
> c) readthedocs.org is an excellent Open Source site that builds and 
> hosts sphinx docs for projects. They have an interface for docs 
> requirements documented and defined that we can align. By aligning, 
> projects can use migrate between docs.o.o and readthedocs.org and still 
> have a consistent experience.
> 
> The PTI I'd like to propose for this is more complex, so I'd like to 
> describe it in terms of:
> 
> - OpenStack organizational requirements
> - helper sugar for developers with per-language recommendations
> 

Re: [openstack-dev] [tc][docs][release] Updating the PTI for docs and tarballs

2017-09-30 Thread Doug Hellmann
Excerpts from Monty Taylor's message of 2017-09-30 10:20:08 -0500:
> Hey everybody,
> 
> Oh goodie, I can hear you say, let's definitely spend some time 
> bikeshedding about specific command invocations related to building docs 
> and tarballs!!!
> 
> tl;dr I want to change the PTI for docs and tarball building to be less 
> OpenStack-specific
> 
> The Problem
> ===
> 
> As we work on Zuul v3, there are a set of job definitions that are 
> rather fundamental that can totally be directly shared between Zuul 
> installations whether those Zuuls are working with OpenStack content or 
> not. As an example "tox -epy27" is a fairly standard thing, so a Zuul 
> job called "tox-py27" has no qualities specific to OpenStack and could 
> realistically be used by anyone who uses tox to manage their project.
> 
> Docs and Tarballs builds for us, however, are the following:
> 
> tox -evenv -- python setup.py sdist
> tox -evenv -- python setup.py build_sphinx
> 
> Neither of those are things that are likely to work outside of 
> OpenStack. (The 'venv' tox environment is not a default tox thing)
> 
> I'm going to argue neither of them are actually providing us with much 
> value.
> 
> Tarball Creation
> 
> 
> Tarball creation is super simple. setup_requires is already handled out 
> of band of everything else. Go clone nova into a completely empty system 
> and run python setup.py sdist ... and it works. (actually, nova is big. 
> use something smaller like gertty ...)
> 
> docker run -it --rm python bash -c 'git clone \
>   https://git.openstack.org/openstack/gertty && cd gertty \
>   && python setup.py sdist'
> 
> There is not much value in that tox wrapper - and it's not like it's 
> making it EASIER to run the command. In fact, it's more typing.
> 
> I propose we change the PTI from:
> 
>tox -evenv python setup.py sdist
> 
> to:
> 
>python setup.py sdist
> 
> and then change the gate jobs to use the non-tox form of the command.
> 
> I'd also like to further change it to be explicit that we also build 
> wheels. So the ACTUAL commands that the project should support are:
> 
>python setup.py sdist
>python setup.py bdist_wheel
> 
> All of our projects support this already, so this should be a no-op.
> 
> Notes:
> 
> * Python projects that need to build C extensions might need their pip 
> requirements (and bindep requirements) installed in order to run 
> bdist_wheel. We do not support that broadly at the moment ANYWAY - so 
> I'd like to leave that as an outlier and handle it when we need to 
> handle it.
> 
> * It's *possible* that somewhere we have a repo that has somehow done 
> something that would cause python setup.py sdist or python setup.py 
> bdist_wheel to not work without pip requirements installed. I believe we 
> should consider that a bug and fix it in the project if we find such a 
> thing - but since we use pbr in all of the OpenStack projects, I find it 
> extremely unlikely.
> 
> Governance patch submitted: https://review.openstack.org/508693
> 
> Sphinx Documentation
> 
> 
> Doc builds are more complex - but I think there is a high amount of 
> value in changing how we invoke them for a few reasons.
> 
> a) nobody uses 'tox -evenv -- python setup.py build_sphinx' but us
> b) we decided to use sphinx for go and javascript - but we invoke sphinx 
> differently for each of those (since they naturally don't have tox), 
> meaning we can't just have a "build-sphinx-docs" job and even share it 
> with ourselves.
> c) readthedocs.org is an excellent Open Source site that builds and 
> hosts sphinx docs for projects. They have an interface for docs 
> requirements documented and defined that we can align. By aligning, 
> projects can use migrate between docs.o.o and readthedocs.org and still 
> have a consistent experience.
> 
> The PTI I'd like to propose for this is more complex, so I'd like to 
> describe it in terms of:
> 
> - OpenStack organizational requirements
> - helper sugar for developers with per-language recommendations
> 
> I believe the result can be a single in-tree doc PTI that applies to 
> python, go and javascript - and a single Zuul job that applies to all of 
> our projects AND non-OpenStack projects as well.
> 
> Organizational Requirements
> ---
> 
> Following readthedocs.org logic we can actually support a wider range of 
> schemes technically, but there is human value in having consistency on 
> these topics across our OpenStack repos.
> 
> * docs live in doc/source
> * Python requirements needed by Sphinx to build the docs live in 
> doc/requirements.txt
> 
> If the project is python:
> 
> * doc/requirements.txt can assume the project will have been installed
> * The following should be set in setup.cfg:
> 
>[build_sphinx]
>source-dir = doc/source
>build-dir = doc/build
> 
> Doing the above allows the following commands to work cleanly in 
> automation no matter what the 

Re: [openstack-dev] [tc][docs][release] Updating the PTI for docs and tarballs

2017-09-30 Thread Jeremy Stanley
On 2017-09-30 10:20:08 -0500 (-0500), Monty Taylor wrote:
[...]
> I want to change the PTI for docs and tarball building to be less
> OpenStack-specific
[...]

Looks great! Anything to make OpenStack less snowflake-like and our
build processes more familiar to developers from other communities.
-- 
Jeremy Stanley


signature.asc
Description: Digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [tc][docs][release] Updating the PTI for docs and tarballs

2017-09-30 Thread Monty Taylor

Hey everybody,

Oh goodie, I can hear you say, let's definitely spend some time 
bikeshedding about specific command invocations related to building docs 
and tarballs!!!


tl;dr I want to change the PTI for docs and tarball building to be less 
OpenStack-specific


The Problem
===

As we work on Zuul v3, there are a set of job definitions that are 
rather fundamental that can totally be directly shared between Zuul 
installations whether those Zuuls are working with OpenStack content or 
not. As an example "tox -epy27" is a fairly standard thing, so a Zuul 
job called "tox-py27" has no qualities specific to OpenStack and could 
realistically be used by anyone who uses tox to manage their project.


Docs and Tarballs builds for us, however, are the following:

tox -evenv -- python setup.py sdist
tox -evenv -- python setup.py build_sphinx

Neither of those are things that are likely to work outside of 
OpenStack. (The 'venv' tox environment is not a default tox thing)


I'm going to argue neither of them are actually providing us with much 
value.


Tarball Creation


Tarball creation is super simple. setup_requires is already handled out 
of band of everything else. Go clone nova into a completely empty system 
and run python setup.py sdist ... and it works. (actually, nova is big. 
use something smaller like gertty ...)


   docker run -it --rm python bash -c 'git clone \
 https://git.openstack.org/openstack/gertty && cd gertty \
 && python setup.py sdist'

There is not much value in that tox wrapper - and it's not like it's 
making it EASIER to run the command. In fact, it's more typing.


I propose we change the PTI from:

  tox -evenv python setup.py sdist

to:

  python setup.py sdist

and then change the gate jobs to use the non-tox form of the command.

I'd also like to further change it to be explicit that we also build 
wheels. So the ACTUAL commands that the project should support are:


  python setup.py sdist
  python setup.py bdist_wheel

All of our projects support this already, so this should be a no-op.

Notes:

* Python projects that need to build C extensions might need their pip 
requirements (and bindep requirements) installed in order to run 
bdist_wheel. We do not support that broadly at the moment ANYWAY - so 
I'd like to leave that as an outlier and handle it when we need to 
handle it.


* It's *possible* that somewhere we have a repo that has somehow done 
something that would cause python setup.py sdist or python setup.py 
bdist_wheel to not work without pip requirements installed. I believe we 
should consider that a bug and fix it in the project if we find such a 
thing - but since we use pbr in all of the OpenStack projects, I find it 
extremely unlikely.


Governance patch submitted: https://review.openstack.org/508693

Sphinx Documentation


Doc builds are more complex - but I think there is a high amount of 
value in changing how we invoke them for a few reasons.


a) nobody uses 'tox -evenv -- python setup.py build_sphinx' but us
b) we decided to use sphinx for go and javascript - but we invoke sphinx 
differently for each of those (since they naturally don't have tox), 
meaning we can't just have a "build-sphinx-docs" job and even share it 
with ourselves.
c) readthedocs.org is an excellent Open Source site that builds and 
hosts sphinx docs for projects. They have an interface for docs 
requirements documented and defined that we can align. By aligning, 
projects can use migrate between docs.o.o and readthedocs.org and still 
have a consistent experience.


The PTI I'd like to propose for this is more complex, so I'd like to 
describe it in terms of:


- OpenStack organizational requirements
- helper sugar for developers with per-language recommendations

I believe the result can be a single in-tree doc PTI that applies to 
python, go and javascript - and a single Zuul job that applies to all of 
our projects AND non-OpenStack projects as well.


Organizational Requirements
---

Following readthedocs.org logic we can actually support a wider range of 
schemes technically, but there is human value in having consistency on 
these topics across our OpenStack repos.


* docs live in doc/source
* Python requirements needed by Sphinx to build the docs live in 
doc/requirements.txt


If the project is python:

* doc/requirements.txt can assume the project will have been installed
* The following should be set in setup.cfg:

  [build_sphinx]
  source-dir = doc/source
  build-dir = doc/build

Doing the above allows the following commands to work cleanly in 
automation no matter what the language is:


  [ -f doc/requirements.txt ] && pip install -rdoc/requirements.txt
  sphinx-build -b doc/source doc/build

No additional commands should be required.

The setup.cfg stanza allows:

  python setup.py build_sphinx

to continue to work. (also, everyone already has one)

Helper sugar for developers