Re: [openstack-dev] [release][infra][zuul][zuulv3][horizon][neutron] project-specific release job templates

[Doug Hellmann]

Excerpts from Doug Hellmann's message of 2017-10-24 09:42:25 -0400:
> The neutron queens milestone release is held up right now because
> some of the repositories are using a release job template that isn't
> recognized by the validation code in the releases repository.  I'm
> trying to decide between adding the custom job template to the
> validation code or changing the release jobs for those neutron
> repositories to use the one that isn't custom. I think we'll have
> the same problem with horizon plugins using the custom job template
> set up for them.
> 
> It looks like the publish-to-pypi-neutron template modifies
> publish-to-pypi by adding openstack/neutron to the required-repositories
> list for the release-openstack-python job. That repository was also at
> some point added directly to the release-openstack-python job. So
> technically the extension via the template is not needed. The same
> applies to publish-to-pypi-horizon.
> 
> I see a few issues with keeping job template variants:
> 
> 1. Having multiple release job variants complicates the release
>repo validation logic. That logic was put in place after we
>discovered several projects without release jobs defined at all,
>so we definitely want to keep some level of validation in place.
> 
> 2. As we continue to make changes to the release jobs, we're going
>to have to consider whether to make those changes in multiple
>places, which seems error prone.
> 
> 3. As we find other projects with more dependencies, we're going
>to end up with more custom templates.
> 
> Those issues may be mitigated if we move the release job definitions
> into the releases repo as we have discussed, because it will be
> more obvious that we have multiple related templates that are
> variants of one another and we can make the relevant changes all
> together in one patch.
> 
> One alternative to keeping multiple variants, and defining more in
> the future, is to add required-repositories to the release-openstack-python
> job directly, as we discover they are needed. Of course that means
> we will clone repositories for some jobs that don't actually use
> them. I don't know how big of an issue that really is, but the issue
> of not knowing which instances of a job actually need a particular
> dependency seems like more of a justification for keeping separate
> templates than any runtime savings we would have by skipping a
> couple of extra calls to git clone.
> 
> It feels like we have two related by not necessarily dependent
> policy questions we need to answer before we decide how to proceed:
> 
> (a) Do we want to move the release job definitions from project-config
> and openstack-zuul-jobs to the releases repo?
> 
> (b) Do we want to have multiple release job templates due to custom
> dependencies (or any other reason)?
> 
> Thoughts?
> 
> Doug
> 

Based on the conversation in this thread and on IRC, we decided to
keep the job variants and update the releases repo so projects can
explicitly indicate that they are using those variants instead of
the default. See https://review.openstack.org/515119 for that change,
and there are related changes in the series for deliverable files
that needed to be updated.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [release][infra][zuul][zuulv3][horizon][neutron] project-specific release job templates

[Jeremy Stanley]

On 2017-10-24 16:24:39 -0700 (-0700), Armando M. wrote:
[...]
> I hope I represented the conversation correctly!
[...]

Totally accurate, thanks!
-- 
Jeremy Stanley


signature.asc
Description: Digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [release][infra][zuul][zuulv3][horizon][neutron] project-specific release job templates

[Armando M.]

On 24 October 2017 at 13:13, Jeremy Stanley  wrote:

> On 2017-10-24 11:31:56 -0700 (-0700), Armando M. wrote:
> [...]
> > the work on neutron-lib is slowly progressing but it's not close enough
> to
> > allow us to break the dependency that requires neutron sub-projects to
> add
> > neutron to the list of required-projects (which I believe the sort of the
> > error in the release pipeline stems from).
> [...]
>
> It's not entirely clear to me what about a Neutron plug-in would
> actually require the neutron source installed just to be able to
> generate a tarball (or wheel) or build sphinx docs. Couldn't that be
> solved even if things like installation or unit testing still
> continue to need neutron?
>

Jeremy and I talked about this in [1].

To sum it up: the necessity of neutron for plugins is primarily for testing
and installation to the best of my knowledge. Other coupling tied to bash
scripts that control gate jobs could be easily addressed if it gets in the
way. Therefore, should we move away from tox for release specific tasks,
it's entirely possible to drop the custom project definition.

I hope I represented the conversation correctly!

[1]
http://eavesdrop.openstack.org/irclogs/%23openstack-infra/latest.log.html#t2017-10-24T23:06:35


> --
> Jeremy Stanley
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [release][infra][zuul][zuulv3][horizon][neutron] project-specific release job templates

[Jeremy Stanley]

On 2017-10-24 11:31:56 -0700 (-0700), Armando M. wrote:
[...]
> the work on neutron-lib is slowly progressing but it's not close enough to
> allow us to break the dependency that requires neutron sub-projects to add
> neutron to the list of required-projects (which I believe the sort of the
> error in the release pipeline stems from).
[...]

It's not entirely clear to me what about a Neutron plug-in would
actually require the neutron source installed just to be able to
generate a tarball (or wheel) or build sphinx docs. Couldn't that be
solved even if things like installation or unit testing still
continue to need neutron?
-- 
Jeremy Stanley


signature.asc
Description: Digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [release][infra][zuul][zuulv3][horizon][neutron] project-specific release job templates

[Armando M.]

On 24 October 2017 at 10:35, Doug Hellmann  wrote:

> Excerpts from Jeremy Stanley's message of 2017-10-24 15:05:34 +:
> > On 2017-10-24 09:42:25 -0400 (-0400), Doug Hellmann wrote:
> > [...]
> > > It looks like the publish-to-pypi-neutron template modifies
> > > publish-to-pypi by adding openstack/neutron to the
> required-repositories
> > > list for the release-openstack-python job. That repository was also at
> > > some point added directly to the release-openstack-python job. So
> > > technically the extension via the template is not needed. The same
> > > applies to publish-to-pypi-horizon.
> > [...]
> >
> > Both are stop-gap solutions and I think either is fine in the short
> > term to get us through the bulk of the release request backlog.
> > Longer term we want to have neither of those. Python projects should
> > be able to build sdists/wheels[1], documentation[2] and release
> > notes[3] without tox (a behavior change which significantly
> > complicates preinstalling source code from other projects, so best
> > if their release jobs simply don't rely on doing that at all).
> >
> > > As we find other projects with more dependencies, we're going to
> > > end up with more custom templates.
> > [...]
> >
> > If we do, this only accelerates the need for something like a
> > community goal for fixing this in those respective Python
> > plugin/extension projects.
>
> Right. I know the neutron team has been working on their library system
> for quite a while. Maybe someone can report on the progress there?
>

the work on neutron-lib is slowly progressing but it's not close enough to
allow us to break the dependency that requires neutron sub-projects to add
neutron to the list of required-projects (which I believe the sort of the
error in the release pipeline stems from).


>
> I don't know if the horizon team is working on that (I'm just uninformed
> about what they're doing). So maybe someone from that team can comment?
>
> >
> > > One alternative to keeping multiple variants, and defining more in
> > > the future, is to add required-repositories to the
> release-openstack-python
> > > job directly, as we discover they are needed. Of course that means
> > > we will clone repositories for some jobs that don't actually use
> > > them. I don't know how big of an issue that really is, but the issue
> > > of not knowing which instances of a job actually need a particular
> > > dependency seems like more of a justification for keeping separate
> > > templates than any runtime savings we would have by skipping a
> > > couple of extra calls to git clone.
> > [...]
> >
> > Well, in this case you're talking about Zuul needing to calculate
> > merge states for the neutron and horizon repos and then push these
> > into every build of the affected jobs, which is no small amount of
> > overhead on its own given the size of those particular repos. Also,
> > once the tox-siblings role[4] is back in action (likely later this
> > week?) Zuul will go back to preinstalling any required-projects from
> > source into tox virtualenvs for these jobs, which is quite a bit
> > more overhead still at that point.
>
> Yes, we don't want that, so I think that means we want to retain the
> multiple project templates for the short-term.


> >
> > [1] http://git.openstack.org/cgit/openstack/governance/commit/?
> id=2678231
> > [2] http://git.openstack.org/cgit/openstack/governance/commit/?
> id=2c0cdd2
> > [3] http://git.openstack.org/cgit/openstack/governance/commit/?
> id=df438a7
> > [4] http://git.openstack.org/cgit/openstack-infra/zuul-jobs/
> tree/roles/tox-siblings
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [release][infra][zuul][zuulv3][horizon][neutron] project-specific release job templates

[Doug Hellmann]

Excerpts from Jeremy Stanley's message of 2017-10-24 15:05:34 +:
> On 2017-10-24 09:42:25 -0400 (-0400), Doug Hellmann wrote:
> [...]
> > It looks like the publish-to-pypi-neutron template modifies
> > publish-to-pypi by adding openstack/neutron to the required-repositories
> > list for the release-openstack-python job. That repository was also at
> > some point added directly to the release-openstack-python job. So
> > technically the extension via the template is not needed. The same
> > applies to publish-to-pypi-horizon.
> [...]
> 
> Both are stop-gap solutions and I think either is fine in the short
> term to get us through the bulk of the release request backlog.
> Longer term we want to have neither of those. Python projects should
> be able to build sdists/wheels[1], documentation[2] and release
> notes[3] without tox (a behavior change which significantly
> complicates preinstalling source code from other projects, so best
> if their release jobs simply don't rely on doing that at all).
> 
> > As we find other projects with more dependencies, we're going to
> > end up with more custom templates.
> [...]
> 
> If we do, this only accelerates the need for something like a
> community goal for fixing this in those respective Python
> plugin/extension projects.

Right. I know the neutron team has been working on their library system
for quite a while. Maybe someone can report on the progress there?

I don't know if the horizon team is working on that (I'm just uninformed
about what they're doing). So maybe someone from that team can comment?

> 
> > One alternative to keeping multiple variants, and defining more in
> > the future, is to add required-repositories to the release-openstack-python
> > job directly, as we discover they are needed. Of course that means
> > we will clone repositories for some jobs that don't actually use
> > them. I don't know how big of an issue that really is, but the issue
> > of not knowing which instances of a job actually need a particular
> > dependency seems like more of a justification for keeping separate
> > templates than any runtime savings we would have by skipping a
> > couple of extra calls to git clone.
> [...]
> 
> Well, in this case you're talking about Zuul needing to calculate
> merge states for the neutron and horizon repos and then push these
> into every build of the affected jobs, which is no small amount of
> overhead on its own given the size of those particular repos. Also,
> once the tox-siblings role[4] is back in action (likely later this
> week?) Zuul will go back to preinstalling any required-projects from
> source into tox virtualenvs for these jobs, which is quite a bit
> more overhead still at that point.

Yes, we don't want that, so I think that means we want to retain the
multiple project templates for the short-term.

> 
> [1] http://git.openstack.org/cgit/openstack/governance/commit/?id=2678231
> [2] http://git.openstack.org/cgit/openstack/governance/commit/?id=2c0cdd2
> [3] http://git.openstack.org/cgit/openstack/governance/commit/?id=df438a7
> [4] 
> http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/tox-siblings

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [release][infra][zuul][zuulv3][horizon][neutron] project-specific release job templates

[Jeremy Stanley]

On 2017-10-24 09:42:25 -0400 (-0400), Doug Hellmann wrote:
[...]
> It looks like the publish-to-pypi-neutron template modifies
> publish-to-pypi by adding openstack/neutron to the required-repositories
> list for the release-openstack-python job. That repository was also at
> some point added directly to the release-openstack-python job. So
> technically the extension via the template is not needed. The same
> applies to publish-to-pypi-horizon.
[...]

Both are stop-gap solutions and I think either is fine in the short
term to get us through the bulk of the release request backlog.
Longer term we want to have neither of those. Python projects should
be able to build sdists/wheels[1], documentation[2] and release
notes[3] without tox (a behavior change which significantly
complicates preinstalling source code from other projects, so best
if their release jobs simply don't rely on doing that at all).

> As we find other projects with more dependencies, we're going to
> end up with more custom templates.
[...]

If we do, this only accelerates the need for something like a
community goal for fixing this in those respective Python
plugin/extension projects.

> One alternative to keeping multiple variants, and defining more in
> the future, is to add required-repositories to the release-openstack-python
> job directly, as we discover they are needed. Of course that means
> we will clone repositories for some jobs that don't actually use
> them. I don't know how big of an issue that really is, but the issue
> of not knowing which instances of a job actually need a particular
> dependency seems like more of a justification for keeping separate
> templates than any runtime savings we would have by skipping a
> couple of extra calls to git clone.
[...]

Well, in this case you're talking about Zuul needing to calculate
merge states for the neutron and horizon repos and then push these
into every build of the affected jobs, which is no small amount of
overhead on its own given the size of those particular repos. Also,
once the tox-siblings role[4] is back in action (likely later this
week?) Zuul will go back to preinstalling any required-projects from
source into tox virtualenvs for these jobs, which is quite a bit
more overhead still at that point.

[1] http://git.openstack.org/cgit/openstack/governance/commit/?id=2678231
[2] http://git.openstack.org/cgit/openstack/governance/commit/?id=2c0cdd2
[3] http://git.openstack.org/cgit/openstack/governance/commit/?id=df438a7
[4] 
http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/tox-siblings
-- 
Jeremy Stanley


signature.asc
Description: Digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [release][infra][zuul][zuulv3][horizon][neutron] project-specific release job templates

[Davanum Srinivas]

Please see below:

On Tue, Oct 24, 2017 at 10:11 AM, Thierry Carrez  wrote:
> Doug Hellmann wrote:
>> [...]
>> It feels like we have two related by not necessarily dependent
>> policy questions we need to answer before we decide how to proceed:
>
> I don't have strong opinions on any of those two, I think anything would
> work. If I had to pick between yes and no, here is my gut feeling:
>
>> (a) Do we want to move the release job definitions from project-config
>> and openstack-zuul-jobs to the releases repo?
>
> Leaning towards yes as it will (imho) simplify maintenance. We'll still
> have a couple of privileged jobs that will live on the infra side though?

Agree with Thierry.

>> (b) Do we want to have multiple release job templates due to custom
>> dependencies (or any other reason)?
>
> Leaning towards no if we can avoid it, again to simplify operations.
> That said, it will be difficult to enforce...

Same. Let's start without it and revisit if we need them later.

> --
> Thierry Carrez (ttx)
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [release][infra][zuul][zuulv3][horizon][neutron] project-specific release job templates

[Sean McGinnis]

On Tue, Oct 24, 2017 at 09:42:25AM -0400, Doug Hellmann wrote:
> The neutron queens milestone release is held up right now because
> some of the repositories are using a release job template that isn't
> recognized by the validation code in the releases repository.  I'm
> trying to decide between adding the custom job template to the
> validation code or changing the release jobs for those neutron
> repositories to use the one that isn't custom. I think we'll have
> the same problem with horizon plugins using the custom job template
> set up for them.
> 
> It looks like the publish-to-pypi-neutron template modifies
> publish-to-pypi by adding openstack/neutron to the required-repositories
> list for the release-openstack-python job. That repository was also at
> some point added directly to the release-openstack-python job. So
> technically the extension via the template is not needed. The same
> applies to publish-to-pypi-horizon.
> 
> I see a few issues with keeping job template variants:
> 
> 1. Having multiple release job variants complicates the release
>repo validation logic. That logic was put in place after we
>discovered several projects without release jobs defined at all,
>so we definitely want to keep some level of validation in place.
> 
> 2. As we continue to make changes to the release jobs, we're going
>to have to consider whether to make those changes in multiple
>places, which seems error prone.

This is my big concern. Chances are very likely things will get missed.

> 
> 3. As we find other projects with more dependencies, we're going
>to end up with more custom templates.
> 
> Those issues may be mitigated if we move the release job definitions
> into the releases repo as we have discussed, because it will be
> more obvious that we have multiple related templates that are
> variants of one another and we can make the relevant changes all
> together in one patch.
> 
> One alternative to keeping multiple variants, and defining more in
> the future, is to add required-repositories to the release-openstack-python
> job directly, as we discover they are needed. Of course that means
> we will clone repositories for some jobs that don't actually use
> them. I don't know how big of an issue that really is, but the issue
> of not knowing which instances of a job actually need a particular
> dependency seems like more of a justification for keeping separate
> templates than any runtime savings we would have by skipping a
> couple of extra calls to git clone.
> 
> It feels like we have two related by not necessarily dependent
> policy questions we need to answer before we decide how to proceed:
> 
> (a) Do we want to move the release job definitions from project-config
> and openstack-zuul-jobs to the releases repo?
> 

This seems like a good compromise to me. If we at least have them all in one
place, it will make it a lot easier to make sure we are able to get everything
updated as needed.

> (b) Do we want to have multiple release job templates due to custom
> dependencies (or any other reason)?
> 

I'm guessing we will want this. We will most likely at some point have some
need to have unique requirements for certain jobs.

> Thoughts?
> 
> Doug
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [release][infra][zuul][zuulv3][horizon][neutron] project-specific release job templates

[Thierry Carrez]

Doug Hellmann wrote:
> [...]
> It feels like we have two related by not necessarily dependent
> policy questions we need to answer before we decide how to proceed:

I don't have strong opinions on any of those two, I think anything would
work. If I had to pick between yes and no, here is my gut feeling:

> (a) Do we want to move the release job definitions from project-config
> and openstack-zuul-jobs to the releases repo?

Leaning towards yes as it will (imho) simplify maintenance. We'll still
have a couple of privileged jobs that will live on the infra side though?

> (b) Do we want to have multiple release job templates due to custom
> dependencies (or any other reason)?

Leaning towards no if we can avoid it, again to simplify operations.
That said, it will be difficult to enforce...

-- 
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [release][infra][zuul][zuulv3][horizon][neutron] project-specific release job templates

[Doug Hellmann]

The neutron queens milestone release is held up right now because
some of the repositories are using a release job template that isn't
recognized by the validation code in the releases repository.  I'm
trying to decide between adding the custom job template to the
validation code or changing the release jobs for those neutron
repositories to use the one that isn't custom. I think we'll have
the same problem with horizon plugins using the custom job template
set up for them.

It looks like the publish-to-pypi-neutron template modifies
publish-to-pypi by adding openstack/neutron to the required-repositories
list for the release-openstack-python job. That repository was also at
some point added directly to the release-openstack-python job. So
technically the extension via the template is not needed. The same
applies to publish-to-pypi-horizon.

I see a few issues with keeping job template variants:

1. Having multiple release job variants complicates the release
   repo validation logic. That logic was put in place after we
   discovered several projects without release jobs defined at all,
   so we definitely want to keep some level of validation in place.

2. As we continue to make changes to the release jobs, we're going
   to have to consider whether to make those changes in multiple
   places, which seems error prone.

3. As we find other projects with more dependencies, we're going
   to end up with more custom templates.

Those issues may be mitigated if we move the release job definitions
into the releases repo as we have discussed, because it will be
more obvious that we have multiple related templates that are
variants of one another and we can make the relevant changes all
together in one patch.

One alternative to keeping multiple variants, and defining more in
the future, is to add required-repositories to the release-openstack-python
job directly, as we discover they are needed. Of course that means
we will clone repositories for some jobs that don't actually use
them. I don't know how big of an issue that really is, but the issue
of not knowing which instances of a job actually need a particular
dependency seems like more of a justification for keeping separate
templates than any runtime savings we would have by skipping a
couple of extra calls to git clone.

It feels like we have two related by not necessarily dependent
policy questions we need to answer before we decide how to proceed:

(a) Do we want to move the release job definitions from project-config
and openstack-zuul-jobs to the releases repo?

(b) Do we want to have multiple release job templates due to custom
dependencies (or any other reason)?

Thoughts?

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev