Re: [openstack-dev] [gate] any project using olso.db test_migrations is currently blocked

[Jay Pipes]

On 12/16/2015 01:03 PM, Carl Baldwin wrote:

On Wed, Dec 16, 2015 at 10:04 AM, Mike Bayer  wrote:

Instead of just breaking the world, and burning 10s to 100 engineer
hours in redo tests and investigating and addressing the break after the
fact.


We shouldn't let this happen in the first place.  I think this is our fault.

We need to vet new package releases before they wreak havoc.  We need
to accept new package releases by proposing a patch to update the
version and take it through the gate.  Weren't we working on this at
one point?  I understand if it isn't quite possible to do this yet but
we need to be working toward this and accelerating our efforts rather
than lashing out at package maintainers.

With the scale that openstack development has grown to, we can't
afford to let package updates do this to us.  If we have a patch to
propose accepting new versions, we could provide feedback to package
maintainers in a much more civilized and pleasant way.


I was hoping to get a thanks for even *testing* unreleased versions of
my entirely non-Openstack, upstream projects against Openstack itself.
  If I did *less* effort here, and just didn't bother the way 100% of all
other non-Openstack projects do, then I'd not have been scolded by you.


Requiring any package maintainer to be on top of every consumer of his
package doesn't scale.  We can expect that packages have good tests
and take care to maintain quality and a stability.  But to put the
responsibility for "100s of engineering hours" squarely on Mike's
shoulders because he didn't "raise it up the flagpole" is
irresponsible.  We need to take our own responsibility for the choice
to consume the package and blindly accept updates.


++, well said, Carl.

-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [gate] any project using olso.db test_migrations is currently blocked

[Carl Baldwin]

On Thu, Dec 17, 2015 at 5:29 AM, Ihar Hrachyshka  wrote:
> I will update on Neutron side of things since it seems I have some knowledge
> from the fields.
>
> So first, on resources: we have Sachi and lifeless from infra side + Ihar
> and pc_m looking into expanding constrained jobs in Neutron world.

This is great.  Thank you for your efforts.

> Current status is:
> - all devstack based jobs are already constrained thanks to infra;
> - openstack/neutron pep8/unit/doc/cover jobs are already constrained in
> Liberty and master;
> - for *aas repos, pc_m is currently on top of it, following up on my
> original patches to introduce tox targets and make them voting in master
> gate; we also look into backporting targets and gate setup into Liberty.

This is good progress.

> Note that the new alembic release still broke openstack/neutron repo
> (functional job). This is because even though the job uses devstack to
> bootstrap the environment, it still calls tox to start tests, which makes
> the new venv with no constraints applied prepared and used. Same problem
> probably affects fullstack and api jobs since they use similar setup
> approach.

Is there a bug for this?  If not, I'm considering filing a bug with at
least High importance to track this.  I'd be tempted to mark it as
Critical because when this stuff breaks, it generates a Critical bug.
But, since it isn't blocking anyone *at the moment* maybe High is
better.

> So I have closing the remaining loop hole on my personal agenda; but note
> that the agenda is really packed pretty much all the time, hence no time
> guarantees; so if someone has cycles to help making the neutron gate more
> fault proof, don’t hesitate to volunteer, I am extremely happy to help with
> it.

I understand.  We're all getting pulled in many directions.  My own
agenda is so over-full I don't think anyone could depend on me to
follow through with this.  Is there anyone who could volunteer to tie
up some of these loose ends for Neutron?

Carl

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [gate] any project using olso.db test_migrations is currently blocked

[Ihar Hrachyshka]

Carl Baldwin  wrote:

On Thu, Dec 17, 2015 at 5:29 AM, Ihar Hrachyshka   
wrote:
I will update on Neutron side of things since it seems I have some  
knowledge

from the fields.

So first, on resources: we have Sachi and lifeless from infra side + Ihar
and pc_m looking into expanding constrained jobs in Neutron world.


This is great.  Thank you for your efforts.


Current status is:
- all devstack based jobs are already constrained thanks to infra;
- openstack/neutron pep8/unit/doc/cover jobs are already constrained in
Liberty and master;
- for *aas repos, pc_m is currently on top of it, following up on my
original patches to introduce tox targets and make them voting in master
gate; we also look into backporting targets and gate setup into Liberty.


This is good progress.


Note that the new alembic release still broke openstack/neutron repo
(functional job). This is because even though the job uses devstack to
bootstrap the environment, it still calls tox to start tests, which makes
the new venv with no constraints applied prepared and used. Same problem
probably affects fullstack and api jobs since they use similar setup
approach.


Is there a bug for this?  If not, I'm considering filing a bug with at
least High importance to track this.  I'd be tempted to mark it as
Critical because when this stuff breaks, it generates a Critical bug.
But, since it isn't blocking anyone *at the moment* maybe High is
better.


I haven’t reported it. Please do.




So I have closing the remaining loop hole on my personal agenda; but note
that the agenda is really packed pretty much all the time, hence no time
guarantees; so if someone has cycles to help making the neutron gate more
fault proof, don’t hesitate to volunteer, I am extremely happy to help  
with

it.


I understand.  We're all getting pulled in many directions.  My own
agenda is so over-full I don't think anyone could depend on me to
follow through with this.  Is there anyone who could volunteer to tie
up some of these loose ends for Neutron?


Hopefully. :)

Ihar

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [gate] any project using olso.db test_migrations is currently blocked

[Thomas Goirand]

On 12/16/2015 06:04 PM, Mike Bayer wrote:
> 
> 
> On 12/16/2015 11:53 AM, Sean Dague wrote:
>> On 12/16/2015 11:37 AM, Sean Dague wrote:
>>> On 12/16/2015 11:22 AM, Mike Bayer wrote:


 On 12/16/2015 09:10 AM, Sylvain Bauza wrote:
>
>
> Le 16/12/2015 14:59, Sean Dague a écrit :
>> oslo.db test_migrations is using methods for alembic, which changed in
>> the 0.8.4 release. This ends up causing a unit test failure (at least in
>> the Nova case) that looks like this -
>> http://logs.openstack.org/44/258444/1/check/gate-nova-python27/2ed0401/console.html#_2015-12-16_12_20_17_404
>>
>>
>> There is an oslo.db patch out there
>> https://review.openstack.org/#/c/258478 to fix it, but #openstack-oslo
>> has been pretty quiet this morning, so no idea how fast this can get out
>> into a release.
>>
>> -Sean
>>
>
> So, it seems that the issue came when
> https://bitbucket.org/zzzeek/alembic/issues/341 was merged.
> Fortunatelt, Mike seems to have a patch in place for Nova in order to
> fix this https://review.openstack.org/#/c/253859/
>
> I'd suggest an intensive review pass on that one to make sure it's OK.

 do you folks have a best practice suggestion on this?  My patch kind of
 stayed twisting in the wind for a week even though those who read it
 would have seen "hey, this is going to break on Alembic's next minor
 release!"I pinged the important people and all on it, but it still
 got no attention.
>>>
>>> Which people were those? I guess none of us this morning knew this was
>>> going to be an issue and were surprised that 12 hours worth of patches
>>> had all failed.
>>>
>>> -Sean
>>
>> Best practice is send an email to the openstack-dev list:
>>
>> Subject: [all] the following test jobs will be broken by Alembic 0.8.4
>> release
>>
>> The Alembic 0.8.4 release is scheduled on 12/15. When it comes out it
>> will break Nova unit tests on all branches.
>>
>> The following patch will fix master - .
>>
>> You all will need to backport it as well to all branches.
>>
>>
>> Instead of just breaking the world, and burning 10s to 100 engineer
>> hours in redo tests and investigating and addressing the break after the
>> fact.
> 
> I was hoping to get a thanks for even *testing* unreleased versions of
> my entirely non-Openstack, upstream projects against Openstack itself.
>  If I did *less* effort here, and just didn't bother the way 100% of all
> other non-Openstack projects do, then I'd not have been scolded by you.

IMHO, it shouldn't be *tested*, but *gated*. Meaning that such a
disruptive patch should be accepted only when there's a fix in all of
OpenStack (if that is possible, of course, as I don't really know the
details, just this thread...).

Or do you still consider SQLA / Alembic as just a 3rd party lib for
OpenStack? Wouldn't it be nice to have it maintained directly in
OpenStack infra? Your thoughts?

Cheers,

Thomas Goirand (zigo)


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [gate] any project using olso.db test_migrations is currently blocked

[Ihar Hrachyshka]

Jeremy Stanley  wrote:


On 2015-12-16 13:59:55 -0700 (-0700), Carl Baldwin wrote:

Is someone from Neutron actively helping out here?  Need more?

[...]

I believe all of the jobs currently voting on changes proposed to
the master branch of the openstack/neutron repo are using centrally
constrained requirements (they all either have "constraints" or
"dsvm" in their job names).

I'll let Robert and Sachi, who have been spearheading this effort up
to this point, comment on whether additional assistance is needed
and where they see next steps leading.


I will update on Neutron side of things since it seems I have some  
knowledge from the fields.


So first, on resources: we have Sachi and lifeless from infra side + Ihar  
and pc_m looking into expanding constrained jobs in Neutron world.


Current status is:
- all devstack based jobs are already constrained thanks to infra;
- openstack/neutron pep8/unit/doc/cover jobs are already constrained in  
Liberty and master;
- for *aas repos, pc_m is currently on top of it, following up on my  
original patches to introduce tox targets and make them voting in master  
gate; we also look into backporting targets and gate setup into Liberty.


Note that the new alembic release still broke openstack/neutron repo  
(functional job). This is because even though the job uses devstack to  
bootstrap the environment, it still calls tox to start tests, which makes  
the new venv with no constraints applied prepared and used. Same problem  
probably affects fullstack and api jobs since they use similar setup  
approach.


So I have closing the remaining loop hole on my personal agenda; but note  
that the agenda is really packed pretty much all the time, hence no time  
guarantees; so if someone has cycles to help making the neutron gate more  
fault proof, don’t hesitate to volunteer, I am extremely happy to help with  
it.


Ihar

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [gate] any project using olso.db test_migrations is currently blocked

[Mike Bayer]

On 12/17/2015 04:00 PM, Thomas Goirand wrote:
> On 12/16/2015 06:04 PM, Mike Bayer wrote:
>>
>>
>> On 12/16/2015 11:53 AM, Sean Dague wrote:
>>> On 12/16/2015 11:37 AM, Sean Dague wrote:
 On 12/16/2015 11:22 AM, Mike Bayer wrote:
>
>
> On 12/16/2015 09:10 AM, Sylvain Bauza wrote:
>>
>>
>> Le 16/12/2015 14:59, Sean Dague a écrit :
>>> oslo.db test_migrations is using methods for alembic, which changed in
>>> the 0.8.4 release. This ends up causing a unit test failure (at least in
>>> the Nova case) that looks like this -
>>> http://logs.openstack.org/44/258444/1/check/gate-nova-python27/2ed0401/console.html#_2015-12-16_12_20_17_404
>>>
>>>
>>> There is an oslo.db patch out there
>>> https://review.openstack.org/#/c/258478 to fix it, but #openstack-oslo
>>> has been pretty quiet this morning, so no idea how fast this can get out
>>> into a release.
>>>
>>> -Sean
>>>
>>
>> So, it seems that the issue came when
>> https://bitbucket.org/zzzeek/alembic/issues/341 was merged.
>> Fortunatelt, Mike seems to have a patch in place for Nova in order to
>> fix this https://review.openstack.org/#/c/253859/
>>
>> I'd suggest an intensive review pass on that one to make sure it's OK.
>
> do you folks have a best practice suggestion on this?  My patch kind of
> stayed twisting in the wind for a week even though those who read it
> would have seen "hey, this is going to break on Alembic's next minor
> release!"I pinged the important people and all on it, but it still
> got no attention.

 Which people were those? I guess none of us this morning knew this was
 going to be an issue and were surprised that 12 hours worth of patches
 had all failed.

-Sean
>>>
>>> Best practice is send an email to the openstack-dev list:
>>>
>>> Subject: [all] the following test jobs will be broken by Alembic 0.8.4
>>> release
>>>
>>> The Alembic 0.8.4 release is scheduled on 12/15. When it comes out it
>>> will break Nova unit tests on all branches.
>>>
>>> The following patch will fix master - .
>>>
>>> You all will need to backport it as well to all branches.
>>>
>>>
>>> Instead of just breaking the world, and burning 10s to 100 engineer
>>> hours in redo tests and investigating and addressing the break after the
>>> fact.
>>
>> I was hoping to get a thanks for even *testing* unreleased versions of
>> my entirely non-Openstack, upstream projects against Openstack itself.
>>  If I did *less* effort here, and just didn't bother the way 100% of all
>> other non-Openstack projects do, then I'd not have been scolded by you.
> 
> IMHO, it shouldn't be *tested*, but *gated*. Meaning that such a
> disruptive patch should be accepted only when there's a fix in all of
> OpenStack (if that is possible, of course, as I don't really know the
> details, just this thread...).
> 
> Or do you still consider SQLA / Alembic as just a 3rd party lib for
> OpenStack? Wouldn't it be nice to have it maintained directly in
> OpenStack infra? Your thoughts?

Alembic / SQLAlchemy are completely outside of Openstack and are
intrinsic to thousands of non-Openstack environments and userbases.  I
wonder why don't we ask the same question of other Openstack
dependencies, like numpy, lxml, boto, requests, rabbitMQ, and everything
else.

As far as it being *gated*, that is already the plan within Openstack
itself via the upper-constraints system discussed in this thread, which
I mistakenly thought was already in use across the board.  That is, new
release of library X hits pypi, some series of CI only involved with
testing new releases of libs above that of upper-constraints runs tests
on it to see if it breaks current openstack applications, and if so, the
constraints file stays unchanged and the bulk of gate jobs remain
unaffected.



> 
> Cheers,
> 
> Thomas Goirand (zigo)
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [gate] any project using olso.db test_migrations is currently blocked

[Robert Collins]

On 18 December 2015 at 16:58, Mike Bayer  wrote:
>
>

>> Or do you still consider SQLA / Alembic as just a 3rd party lib for
>> OpenStack? Wouldn't it be nice to have it maintained directly in
>> OpenStack infra? Your thoughts?
>
> Alembic / SQLAlchemy are completely outside of Openstack and are
> intrinsic to thousands of non-Openstack environments and userbases.  I
> wonder why don't we ask the same question of other Openstack
> dependencies, like numpy, lxml, boto, requests, rabbitMQ, and everything
> else.

Whats happening organically is that many contributing orgs also
contribute to projects like libvirt, sqlaclhemy and so on - so there's
some 'common funding source' pattern happening - but IMO its entirely
appropriate to consider these projects as independent. Because.. they
are :).

> As far as it being *gated*, that is already the plan within Openstack
> itself via the upper-constraints system discussed in this thread, which
> I mistakenly thought was already in use across the board.  That is, new
> release of library X hits pypi, some series of CI only involved with
> testing new releases of libs above that of upper-constraints runs tests
> on it to see if it breaks current openstack applications, and if so, the
> constraints file stays unchanged and the bulk of gate jobs remain
> unaffected.

Yah, we're rolling that out more broadly at the moment. cookiecutter
has been updated, there's a review setting -constraints as the
expected pattern in the infra manual now.

-Rob


-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [gate] any project using olso.db test_migrations is currently blocked

[Carl Baldwin]

On Wed, Dec 16, 2015 at 11:32 AM, Jeremy Stanley  wrote:
[...]
> Yes, it's progressing nicely. DevStack-based jobs are already
> covered this way for master and stable/liberty, and Neutron is
> piloting the same solution for its other non-DevStack-based jobs. If

Is someone from Neutron actively helping out here?  Need more?

> Nova's unit test jobs were already switched to their
> upper-constraints equivalents then there's a chance this wouldn't
> have impacted there (though we still need to work out the bit where
> we run a representative sample of jobs like neutron/nova unit tests
> on proposed constraints bumps to block any with this sort of impact,
> right now we're really just relying on
> devstack-tempest/grenade/other integration test jobs as canaries).
>
> Anyway, the solution seems to be working (modulo unforeseen issues
> like people thinking it's sane to delete their latest releases of
> some dependencies from PyPI) but it's a long road to full
> implementation.

Thanks for the report Jeremy.  I'm very happy to see progress here.

Carl

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [gate] any project using olso.db test_migrations is currently blocked

[Dan Smith]

> It was in the queue for 11 days, Dan Smith took a look, he added Jay
> Pipes, and I also added Matt Riedemann, there were also a bunch of
> neutron folks on it since this fix originated from their end.

Yeah, and both of those other people have had serious other commitments
in the last two weeks. The commit message in that patch set was a far
cry from "merge this or we break when 0.8.x is released".

Please, in the future, if you know you're going to destroy a half day's
worth of work because a patch isn't getting looked at, spend a little
more effort trying to raise it up the flagpole. Like Sean said, an email
to the list will get serious attention (as this one did), or an agenda
item on the weekly meeting will _certainly_ get enough eyes on the
problem to get it handled.

--Dan

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [gate] any project using olso.db test_migrations is currently blocked

[Flavio Percoco]

On 16/12/15 11:53 -0500, Sean Dague wrote:

On 12/16/2015 11:37 AM, Sean Dague wrote:

On 12/16/2015 11:22 AM, Mike Bayer wrote:



On 12/16/2015 09:10 AM, Sylvain Bauza wrote:



Le 16/12/2015 14:59, Sean Dague a écrit :

oslo.db test_migrations is using methods for alembic, which changed in
the 0.8.4 release. This ends up causing a unit test failure (at least in
the Nova case) that looks like this -
http://logs.openstack.org/44/258444/1/check/gate-nova-python27/2ed0401/console.html#_2015-12-16_12_20_17_404


There is an oslo.db patch out there
https://review.openstack.org/#/c/258478 to fix it, but #openstack-oslo
has been pretty quiet this morning, so no idea how fast this can get out
into a release.

-Sean



So, it seems that the issue came when
https://bitbucket.org/zzzeek/alembic/issues/341 was merged.
Fortunatelt, Mike seems to have a patch in place for Nova in order to
fix this https://review.openstack.org/#/c/253859/

I'd suggest an intensive review pass on that one to make sure it's OK.


do you folks have a best practice suggestion on this?  My patch kind of
stayed twisting in the wind for a week even though those who read it
would have seen "hey, this is going to break on Alembic's next minor
release!"I pinged the important people and all on it, but it still
got no attention.


Which people were those? I guess none of us this morning knew this was
going to be an issue and were surprised that 12 hours worth of patches
had all failed.

-Sean


Best practice is send an email to the openstack-dev list:

Subject: [all] the following test jobs will be broken by Alembic 0.8.4
release

The Alembic 0.8.4 release is scheduled on 12/15. When it comes out it
will break Nova unit tests on all branches.

The following patch will fix master - .

You all will need to backport it as well to all branches.


Instead of just breaking the world, and burning 10s to 100 engineer
hours in redo tests and investigating and addressing the break after the
fact.


I know you didn't want to come off harsh but I think there are better
ways to express recommendations and best practices than this. I don't
think this is the best way to communicate with other members of the
community, especially when they are asking for feedback in good faith,
regardless of how bad the breakage was or how ugly/untolerable the
mistake could've been.

Other than that, I think sending an email out to raise awareness is
probably the best thing to do in these cases and what's normally been
done in the past.

Flavio

--
@flaper87
Flavio Percoco

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [gate] any project using olso.db test_migrations is currently blocked

[Julien Danjou]

On Wed, Dec 16 2015, Carl Baldwin wrote:

> We need to vet new package releases before they wreak havoc.  We need
> to accept new package releases by proposing a patch to update the
> version and take it through the gate.  Weren't we working on this at
> one point?  I understand if it isn't quite possible to do this yet but
> we need to be working toward this and accelerating our efforts rather
> than lashing out at package maintainers.

Projects need to take care of patches that are sent to avoid such things
in the first place.

Blocking new packages (upper cap and all) is just a proof that the
projects fail to cope with their development rate. They should address
that with proper means. Not by blocking the situation and burying itself
under more technical debt. Which seems to be a chronic reflex in many
OpenStack projects.

-- 
Julien Danjou
# Free Software hacker
# https://julien.danjou.info


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [gate] any project using olso.db test_migrations is currently blocked

[Kevin Benton]

Yeah. The twisted take-away from this thread is to not test your changes
against openstack so you have plausible deniability. :)

On Wed, Dec 16, 2015 at 9:04 AM, Mike Bayer  wrote:

>
>
> On 12/16/2015 11:53 AM, Sean Dague wrote:
> > On 12/16/2015 11:37 AM, Sean Dague wrote:
> >> On 12/16/2015 11:22 AM, Mike Bayer wrote:
> >>>
> >>>
> >>> On 12/16/2015 09:10 AM, Sylvain Bauza wrote:
> 
> 
>  Le 16/12/2015 14:59, Sean Dague a écrit :
> > oslo.db test_migrations is using methods for alembic, which changed
> in
> > the 0.8.4 release. This ends up causing a unit test failure (at
> least in
> > the Nova case) that looks like this -
> >
> http://logs.openstack.org/44/258444/1/check/gate-nova-python27/2ed0401/console.html#_2015-12-16_12_20_17_404
> >
> >
> > There is an oslo.db patch out there
> > https://review.openstack.org/#/c/258478 to fix it, but
> #openstack-oslo
> > has been pretty quiet this morning, so no idea how fast this can get
> out
> > into a release.
> >
> > -Sean
> >
> 
>  So, it seems that the issue came when
>  https://bitbucket.org/zzzeek/alembic/issues/341 was merged.
>  Fortunatelt, Mike seems to have a patch in place for Nova in order to
>  fix this https://review.openstack.org/#/c/253859/
> 
>  I'd suggest an intensive review pass on that one to make sure it's OK.
> >>>
> >>> do you folks have a best practice suggestion on this?  My patch kind of
> >>> stayed twisting in the wind for a week even though those who read it
> >>> would have seen "hey, this is going to break on Alembic's next minor
> >>> release!"I pinged the important people and all on it, but it still
> >>> got no attention.
> >>
> >> Which people were those? I guess none of us this morning knew this was
> >> going to be an issue and were surprised that 12 hours worth of patches
> >> had all failed.
> >>
> >>  -Sean
> >
> > Best practice is send an email to the openstack-dev list:
> >
> > Subject: [all] the following test jobs will be broken by Alembic 0.8.4
> > release
> >
> > The Alembic 0.8.4 release is scheduled on 12/15. When it comes out it
> > will break Nova unit tests on all branches.
> >
> > The following patch will fix master - .
> >
> > You all will need to backport it as well to all branches.
> >
> >
> > Instead of just breaking the world, and burning 10s to 100 engineer
> > hours in redo tests and investigating and addressing the break after the
> > fact.
>
> I was hoping to get a thanks for even *testing* unreleased versions of
> my entirely non-Openstack, upstream projects against Openstack itself.
>  If I did *less* effort here, and just didn't bother the way 100% of all
> other non-Openstack projects do, then I'd not have been scolded by you.
>
>
>
>
>
>
> >
> >   -Sean
> >
> >
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Kevin Benton
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [gate] any project using olso.db test_migrations is currently blocked

[Mike Bayer]

On 12/16/2015 11:37 AM, Sean Dague wrote:
> On 12/16/2015 11:22 AM, Mike Bayer wrote:
>>
>>
>> On 12/16/2015 09:10 AM, Sylvain Bauza wrote:
>>>
>>>
>>> Le 16/12/2015 14:59, Sean Dague a écrit :
 oslo.db test_migrations is using methods for alembic, which changed in
 the 0.8.4 release. This ends up causing a unit test failure (at least in
 the Nova case) that looks like this -
 http://logs.openstack.org/44/258444/1/check/gate-nova-python27/2ed0401/console.html#_2015-12-16_12_20_17_404


 There is an oslo.db patch out there
 https://review.openstack.org/#/c/258478 to fix it, but #openstack-oslo
 has been pretty quiet this morning, so no idea how fast this can get out
 into a release.

 -Sean

>>>
>>> So, it seems that the issue came when
>>> https://bitbucket.org/zzzeek/alembic/issues/341 was merged.
>>> Fortunatelt, Mike seems to have a patch in place for Nova in order to
>>> fix this https://review.openstack.org/#/c/253859/
>>>
>>> I'd suggest an intensive review pass on that one to make sure it's OK.
>>
>> do you folks have a best practice suggestion on this?  My patch kind of
>> stayed twisting in the wind for a week even though those who read it
>> would have seen "hey, this is going to break on Alembic's next minor
>> release!"I pinged the important people and all on it, but it still
>> got no attention.
> 
> Which people were those? I guess none of us this morning knew this was
> going to be an issue and were surprised that 12 hours worth of patches
> had all failed.

It was in the queue for 11 days, Dan Smith took a look, he added Jay
Pipes, and I also added Matt Riedemann, there were also a bunch of
neutron folks on it since this fix originated from their end.







> 
>   -Sean
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [gate] any project using olso.db test_migrations is currently blocked

[Carl Baldwin]

On Wed, Dec 16, 2015 at 10:04 AM, Mike Bayer  wrote:
>> Instead of just breaking the world, and burning 10s to 100 engineer
>> hours in redo tests and investigating and addressing the break after the
>> fact.

We shouldn't let this happen in the first place.  I think this is our fault.

We need to vet new package releases before they wreak havoc.  We need
to accept new package releases by proposing a patch to update the
version and take it through the gate.  Weren't we working on this at
one point?  I understand if it isn't quite possible to do this yet but
we need to be working toward this and accelerating our efforts rather
than lashing out at package maintainers.

With the scale that openstack development has grown to, we can't
afford to let package updates do this to us.  If we have a patch to
propose accepting new versions, we could provide feedback to package
maintainers in a much more civilized and pleasant way.

> I was hoping to get a thanks for even *testing* unreleased versions of
> my entirely non-Openstack, upstream projects against Openstack itself.
>  If I did *less* effort here, and just didn't bother the way 100% of all
> other non-Openstack projects do, then I'd not have been scolded by you.

Requiring any package maintainer to be on top of every consumer of his
package doesn't scale.  We can expect that packages have good tests
and take care to maintain quality and a stability.  But to put the
responsibility for "100s of engineering hours" squarely on Mike's
shoulders because he didn't "raise it up the flagpole" is
irresponsible.  We need to take our own responsibility for the choice
to consume the package and blindly accept updates.

Carl

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [gate] any project using olso.db test_migrations is currently blocked

[Mike Bayer]

On 12/16/2015 11:53 AM, Sean Dague wrote:
> On 12/16/2015 11:37 AM, Sean Dague wrote:
>> On 12/16/2015 11:22 AM, Mike Bayer wrote:
>>>
>>>
>>> On 12/16/2015 09:10 AM, Sylvain Bauza wrote:


 Le 16/12/2015 14:59, Sean Dague a écrit :
> oslo.db test_migrations is using methods for alembic, which changed in
> the 0.8.4 release. This ends up causing a unit test failure (at least in
> the Nova case) that looks like this -
> http://logs.openstack.org/44/258444/1/check/gate-nova-python27/2ed0401/console.html#_2015-12-16_12_20_17_404
>
>
> There is an oslo.db patch out there
> https://review.openstack.org/#/c/258478 to fix it, but #openstack-oslo
> has been pretty quiet this morning, so no idea how fast this can get out
> into a release.
>
> -Sean
>

 So, it seems that the issue came when
 https://bitbucket.org/zzzeek/alembic/issues/341 was merged.
 Fortunatelt, Mike seems to have a patch in place for Nova in order to
 fix this https://review.openstack.org/#/c/253859/

 I'd suggest an intensive review pass on that one to make sure it's OK.
>>>
>>> do you folks have a best practice suggestion on this?  My patch kind of
>>> stayed twisting in the wind for a week even though those who read it
>>> would have seen "hey, this is going to break on Alembic's next minor
>>> release!"I pinged the important people and all on it, but it still
>>> got no attention.
>>
>> Which people were those? I guess none of us this morning knew this was
>> going to be an issue and were surprised that 12 hours worth of patches
>> had all failed.
>>
>>  -Sean
> 
> Best practice is send an email to the openstack-dev list:
> 
> Subject: [all] the following test jobs will be broken by Alembic 0.8.4
> release
> 
> The Alembic 0.8.4 release is scheduled on 12/15. When it comes out it
> will break Nova unit tests on all branches.
> 
> The following patch will fix master - .
> 
> You all will need to backport it as well to all branches.
> 
> 
> Instead of just breaking the world, and burning 10s to 100 engineer
> hours in redo tests and investigating and addressing the break after the
> fact.

I was hoping to get a thanks for even *testing* unreleased versions of
my entirely non-Openstack, upstream projects against Openstack itself.
 If I did *less* effort here, and just didn't bother the way 100% of all
other non-Openstack projects do, then I'd not have been scolded by you.






> 
>   -Sean
> 
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [gate] any project using olso.db test_migrations is currently blocked

[Joshua Harlow]

Carl Baldwin wrote:

On Wed, Dec 16, 2015 at 10:04 AM, Mike Bayer  wrote:

Instead of just breaking the world, and burning 10s to 100 engineer
hours in redo tests and investigating and addressing the break after the
fact.


We shouldn't let this happen in the first place.  I think this is our fault.

We need to vet new package releases before they wreak havoc.  We need
to accept new package releases by proposing a patch to update the
version and take it through the gate.  Weren't we working on this at
one point?  I understand if it isn't quite possible to do this yet but
we need to be working toward this and accelerating our efforts rather
than lashing out at package maintainers.

With the scale that openstack development has grown to, we can't
afford to let package updates do this to us.  If we have a patch to
propose accepting new versions, we could provide feedback to package
maintainers in a much more civilized and pleasant way.


I was hoping to get a thanks for even *testing* unreleased versions of
my entirely non-Openstack, upstream projects against Openstack itself.
  If I did *less* effort here, and just didn't bother the way 100% of all
other non-Openstack projects do, then I'd not have been scolded by you.


Requiring any package maintainer to be on top of every consumer of his
package doesn't scale.  We can expect that packages have good tests
and take care to maintain quality and a stability.  But to put the
responsibility for "100s of engineering hours" squarely on Mike's
shoulders because he didn't "raise it up the flagpole" is
irresponsible.  We need to take our own responsibility for the choice
to consume the package and blindly accept updates.


+100 treating people that provide underlying components of openstack 
that everyone depends upon in this way seems u, sorta evil... Mike 
(and many others, in oslo, outside, ...) have built things that many 
many others in our community have built on-top of; so in the spirit of 
xmas (and well being good humans in general) and all that I'd like to 
say thank you mike for your hard work and thank you mike for what u have 
helped do that others have built on. We should all be grateful IMHO for 
the things we don't have to build ourselves that others have built for 
us... (saving us pain, problems, bad APIs... and much much more).




Carl

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [gate] any project using olso.db test_migrations is currently blocked

[Davanum Srinivas]

Mike,

I for one whole heartedly thank you for everything you do including extra nova 
review in this case. 

-- dims


> On Dec 16, 2015, at 12:04 PM, Mike Bayer  wrote:
> 
> 
> 
>> On 12/16/2015 11:53 AM, Sean Dague wrote:
>>> On 12/16/2015 11:37 AM, Sean Dague wrote:
 On 12/16/2015 11:22 AM, Mike Bayer wrote:
 
 
> On 12/16/2015 09:10 AM, Sylvain Bauza wrote:
> 
> 
> Le 16/12/2015 14:59, Sean Dague a écrit :
>> oslo.db test_migrations is using methods for alembic, which changed in
>> the 0.8.4 release. This ends up causing a unit test failure (at least in
>> the Nova case) that looks like this -
>> http://logs.openstack.org/44/258444/1/check/gate-nova-python27/2ed0401/console.html#_2015-12-16_12_20_17_404
>> 
>> 
>> There is an oslo.db patch out there
>> https://review.openstack.org/#/c/258478 to fix it, but #openstack-oslo
>> has been pretty quiet this morning, so no idea how fast this can get out
>> into a release.
>> 
>>-Sean
> 
> So, it seems that the issue came when
> https://bitbucket.org/zzzeek/alembic/issues/341 was merged.
> Fortunatelt, Mike seems to have a patch in place for Nova in order to
> fix this https://review.openstack.org/#/c/253859/
> 
> I'd suggest an intensive review pass on that one to make sure it's OK.
 
 do you folks have a best practice suggestion on this?  My patch kind of
 stayed twisting in the wind for a week even though those who read it
 would have seen "hey, this is going to break on Alembic's next minor
 release!"I pinged the important people and all on it, but it still
 got no attention.
>>> 
>>> Which people were those? I guess none of us this morning knew this was
>>> going to be an issue and were surprised that 12 hours worth of patches
>>> had all failed.
>>> 
>>>-Sean
>> 
>> Best practice is send an email to the openstack-dev list:
>> 
>> Subject: [all] the following test jobs will be broken by Alembic 0.8.4
>> release
>> 
>> The Alembic 0.8.4 release is scheduled on 12/15. When it comes out it
>> will break Nova unit tests on all branches.
>> 
>> The following patch will fix master - .
>> 
>> You all will need to backport it as well to all branches.
>> 
>> 
>> Instead of just breaking the world, and burning 10s to 100 engineer
>> hours in redo tests and investigating and addressing the break after the
>> fact.
> 
> I was hoping to get a thanks for even *testing* unreleased versions of
> my entirely non-Openstack, upstream projects against Openstack itself.
> If I did *less* effort here, and just didn't bother the way 100% of all
> other non-Openstack projects do, then I'd not have been scolded by you.
> 
> 
> 
> 
> 
> 
>> 
>>-Sean
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [gate] any project using olso.db test_migrations is currently blocked

[Sean Dague]

On 12/16/2015 11:37 AM, Sean Dague wrote:
> On 12/16/2015 11:22 AM, Mike Bayer wrote:
>>
>>
>> On 12/16/2015 09:10 AM, Sylvain Bauza wrote:
>>>
>>>
>>> Le 16/12/2015 14:59, Sean Dague a écrit :
 oslo.db test_migrations is using methods for alembic, which changed in
 the 0.8.4 release. This ends up causing a unit test failure (at least in
 the Nova case) that looks like this -
 http://logs.openstack.org/44/258444/1/check/gate-nova-python27/2ed0401/console.html#_2015-12-16_12_20_17_404


 There is an oslo.db patch out there
 https://review.openstack.org/#/c/258478 to fix it, but #openstack-oslo
 has been pretty quiet this morning, so no idea how fast this can get out
 into a release.

 -Sean

>>>
>>> So, it seems that the issue came when
>>> https://bitbucket.org/zzzeek/alembic/issues/341 was merged.
>>> Fortunatelt, Mike seems to have a patch in place for Nova in order to
>>> fix this https://review.openstack.org/#/c/253859/
>>>
>>> I'd suggest an intensive review pass on that one to make sure it's OK.
>>
>> do you folks have a best practice suggestion on this?  My patch kind of
>> stayed twisting in the wind for a week even though those who read it
>> would have seen "hey, this is going to break on Alembic's next minor
>> release!"I pinged the important people and all on it, but it still
>> got no attention.
> 
> Which people were those? I guess none of us this morning knew this was
> going to be an issue and were surprised that 12 hours worth of patches
> had all failed.
> 
>   -Sean

Best practice is send an email to the openstack-dev list:

Subject: [all] the following test jobs will be broken by Alembic 0.8.4
release

The Alembic 0.8.4 release is scheduled on 12/15. When it comes out it
will break Nova unit tests on all branches.

The following patch will fix master - .

You all will need to backport it as well to all branches.


Instead of just breaking the world, and burning 10s to 100 engineer
hours in redo tests and investigating and addressing the break after the
fact.

-Sean


-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [gate] any project using olso.db test_migrations is currently blocked

[Jeremy Stanley]

On 2015-12-16 11:03:38 -0700 (-0700), Carl Baldwin wrote:
[...]
> We need to vet new package releases before they wreak havoc.  We need
> to accept new package releases by proposing a patch to update the
> version and take it through the gate.  Weren't we working on this at
> one point?  I understand if it isn't quite possible to do this yet but
> we need to be working toward this and accelerating our efforts rather
> than lashing out at package maintainers.
[...]

Yes, it's progressing nicely. DevStack-based jobs are already
covered this way for master and stable/liberty, and Neutron is
piloting the same solution for its other non-DevStack-based jobs. If
Nova's unit test jobs were already switched to their
upper-constraints equivalents then there's a chance this wouldn't
have impacted there (though we still need to work out the bit where
we run a representative sample of jobs like neutron/nova unit tests
on proposed constraints bumps to block any with this sort of impact,
right now we're really just relying on
devstack-tempest/grenade/other integration test jobs as canaries).

Anyway, the solution seems to be working (modulo unforeseen issues
like people thinking it's sane to delete their latest releases of
some dependencies from PyPI) but it's a long road to full
implementation.
-- 
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [gate] any project using olso.db test_migrations is currently blocked

[Jeremy Stanley]

On 2015-12-16 16:50:39 -0500 (-0500), Mike Bayer wrote:
[...]
> That said, as the upper-constraints system is available, and not using
> it means that any upstream package release that breaks any test will
> cause 12 hours of gate downtime, I'm surprised that rolling this system
> out across the board isn't an emergency priority. Because right now,
> literally any of the 50 or so projects I see in Nova requirements.txt
> can release something on Pypi at any moment and cause another 12 hours
> of downtime.
[...]

Yep, it happens pretty continually and has for years. The
constraints-based solution is certainly a cross-project priority
effort, but the developers who are in the process of implementing it
are wary of rushing it into place and causing new problems.
Sometimes the devil you know is easier to deal with than the one you
don't.
-- 
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [gate] any project using olso.db test_migrations is currently blocked

[Jeremy Stanley]

On 2015-12-16 13:59:55 -0700 (-0700), Carl Baldwin wrote:
> Is someone from Neutron actively helping out here?  Need more?
[...]

I believe all of the jobs currently voting on changes proposed to
the master branch of the openstack/neutron repo are using centrally
constrained requirements (they all either have "constraints" or
"dsvm" in their job names).

I'll let Robert and Sachi, who have been spearheading this effort up
to this point, comment on whether additional assistance is needed
and where they see next steps leading.
-- 
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [gate] any project using olso.db test_migrations is currently blocked

[Mike Bayer]

On 12/16/2015 01:32 PM, Jeremy Stanley wrote:
> On 2015-12-16 11:03:38 -0700 (-0700), Carl Baldwin wrote:
> [...]
>> We need to vet new package releases before they wreak havoc.  We need
>> to accept new package releases by proposing a patch to update the
>> version and take it through the gate.  Weren't we working on this at
>> one point?  I understand if it isn't quite possible to do this yet but
>> we need to be working toward this and accelerating our efforts rather
>> than lashing out at package maintainers.
> [...]
> 
> Yes, it's progressing nicely. DevStack-based jobs are already
> covered this way for master and stable/liberty, and Neutron is
> piloting the same solution for its other non-DevStack-based jobs. If
> Nova's unit test jobs were already switched to their
> upper-constraints equivalents then there's a chance this wouldn't
> have impacted there (though we still need to work out the bit where
> we run a representative sample of jobs like neutron/nova unit tests
> on proposed constraints bumps to block any with this sort of impact,
> right now we're really just relying on
> devstack-tempest/grenade/other integration test jobs as canaries).
> 
> Anyway, the solution seems to be working (modulo unforeseen issues
> like people thinking it's sane to delete their latest releases of
> some dependencies from PyPI) but it's a long road to full
> implementation.

So just FTR I mistakenly thought that the upper-constraints system was
in place for all gate jobs, and that the release of Alembic 0.8.4 would
at worst have raised a message somewhere within the system that updates
the upper-constraints file, and prevented the version bump from
proceeding.  Had I known that the raw master gate jobs for Nova were
going to go down overnight, I would not have released Alembic in this way.

That said, as the upper-constraints system is available, and not using
it means that any upstream package release that breaks any test will
cause 12 hours of gate downtime, I'm surprised that rolling this system
out across the board isn't an emergency priority. Because right now,
literally any of the 50 or so projects I see in Nova requirements.txt
can release something on Pypi at any moment and cause another 12 hours
of downtime.   Lots of them don't even have major version upper bounds,
even complex and intricate dependencies like lxml and boto.   If a
single test breakage due to an incompatible change in an upstream
release means 100 man hours lost and 12 hours of downtime, then Nova/
others are essentially a giant boulder balanced on the edge of a cliff,
with 50 or so loaded BB guns on Pypi pointed at it.








> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [gate] any project using olso.db test_migrations is currently blocked

[Matt Riedemann]

On 12/16/2015 10:22 AM, Mike Bayer wrote:



On 12/16/2015 09:10 AM, Sylvain Bauza wrote:



Le 16/12/2015 14:59, Sean Dague a écrit :

oslo.db test_migrations is using methods for alembic, which changed in
the 0.8.4 release. This ends up causing a unit test failure (at least in
the Nova case) that looks like this -
http://logs.openstack.org/44/258444/1/check/gate-nova-python27/2ed0401/console.html#_2015-12-16_12_20_17_404


There is an oslo.db patch out there
https://review.openstack.org/#/c/258478 to fix it, but #openstack-oslo
has been pretty quiet this morning, so no idea how fast this can get out
into a release.

 -Sean



So, it seems that the issue came when
https://bitbucket.org/zzzeek/alembic/issues/341 was merged.
Fortunatelt, Mike seems to have a patch in place for Nova in order to
fix this https://review.openstack.org/#/c/253859/

I'd suggest an intensive review pass on that one to make sure it's OK.


do you folks have a best practice suggestion on this?  My patch kind of
stayed twisting in the wind for a week even though those who read it
would have seen "hey, this is going to break on Alembic's next minor
release!"I pinged the important people and all on it, but it still
got no attention.

I thought of adding a launchpad bug but I had the impression that too
would just be more idle webpages sitting there until I just put the
release out, and then the whole thing got attention / fixed in just a
few hours!

I'm not sure there's any other upstream (non openstack/stackforge)
projects that actually run Openstack tests on their own CI against
upcoming versions like I do (the SQLAlchemy project actually spends
money on Amazon EC2 instances that are utilized for running these
suites, among others).









-Sylvain


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



A mailing list post or having it in the nova meeting agenda would have 
gotten attention better than adding people to a gerrit review. Even 
pinging on IRC would be better than just adding people to gerrit.


--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [gate] any project using olso.db test_migrations is currently blocked

[Sylvain Bauza]

Le 16/12/2015 14:59, Sean Dague a écrit :

oslo.db test_migrations is using methods for alembic, which changed in
the 0.8.4 release. This ends up causing a unit test failure (at least in
the Nova case) that looks like this -
http://logs.openstack.org/44/258444/1/check/gate-nova-python27/2ed0401/console.html#_2015-12-16_12_20_17_404

There is an oslo.db patch out there
https://review.openstack.org/#/c/258478 to fix it, but #openstack-oslo
has been pretty quiet this morning, so no idea how fast this can get out
into a release.

-Sean



So, it seems that the issue came when 
https://bitbucket.org/zzzeek/alembic/issues/341 was merged.
Fortunatelt, Mike seems to have a patch in place for Nova in order to 
fix this https://review.openstack.org/#/c/253859/


I'd suggest an intensive review pass on that one to make sure it's OK.

-Sylvain


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [gate] any project using olso.db test_migrations is currently blocked

[Sean Dague]

oslo.db test_migrations is using methods for alembic, which changed in
the 0.8.4 release. This ends up causing a unit test failure (at least in
the Nova case) that looks like this -
http://logs.openstack.org/44/258444/1/check/gate-nova-python27/2ed0401/console.html#_2015-12-16_12_20_17_404

There is an oslo.db patch out there
https://review.openstack.org/#/c/258478 to fix it, but #openstack-oslo
has been pretty quiet this morning, so no idea how fast this can get out
into a release.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [gate] any project using olso.db test_migrations is currently blocked

[Mike Bayer]

On 12/16/2015 09:10 AM, Sylvain Bauza wrote:
> 
> 
> Le 16/12/2015 14:59, Sean Dague a écrit :
>> oslo.db test_migrations is using methods for alembic, which changed in
>> the 0.8.4 release. This ends up causing a unit test failure (at least in
>> the Nova case) that looks like this -
>> http://logs.openstack.org/44/258444/1/check/gate-nova-python27/2ed0401/console.html#_2015-12-16_12_20_17_404
>>
>>
>> There is an oslo.db patch out there
>> https://review.openstack.org/#/c/258478 to fix it, but #openstack-oslo
>> has been pretty quiet this morning, so no idea how fast this can get out
>> into a release.
>>
>> -Sean
>>
> 
> So, it seems that the issue came when
> https://bitbucket.org/zzzeek/alembic/issues/341 was merged.
> Fortunatelt, Mike seems to have a patch in place for Nova in order to
> fix this https://review.openstack.org/#/c/253859/
> 
> I'd suggest an intensive review pass on that one to make sure it's OK.

do you folks have a best practice suggestion on this?  My patch kind of
stayed twisting in the wind for a week even though those who read it
would have seen "hey, this is going to break on Alembic's next minor
release!"I pinged the important people and all on it, but it still
got no attention.

I thought of adding a launchpad bug but I had the impression that too
would just be more idle webpages sitting there until I just put the
release out, and then the whole thing got attention / fixed in just a
few hours!

I'm not sure there's any other upstream (non openstack/stackforge)
projects that actually run Openstack tests on their own CI against
upcoming versions like I do (the SQLAlchemy project actually spends
money on Amazon EC2 instances that are utilized for running these
suites, among others).







> 
> -Sylvain
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [gate] any project using olso.db test_migrations is currently blocked

[Sean Dague]

On 12/16/2015 11:22 AM, Mike Bayer wrote:
> 
> 
> On 12/16/2015 09:10 AM, Sylvain Bauza wrote:
>>
>>
>> Le 16/12/2015 14:59, Sean Dague a écrit :
>>> oslo.db test_migrations is using methods for alembic, which changed in
>>> the 0.8.4 release. This ends up causing a unit test failure (at least in
>>> the Nova case) that looks like this -
>>> http://logs.openstack.org/44/258444/1/check/gate-nova-python27/2ed0401/console.html#_2015-12-16_12_20_17_404
>>>
>>>
>>> There is an oslo.db patch out there
>>> https://review.openstack.org/#/c/258478 to fix it, but #openstack-oslo
>>> has been pretty quiet this morning, so no idea how fast this can get out
>>> into a release.
>>>
>>> -Sean
>>>
>>
>> So, it seems that the issue came when
>> https://bitbucket.org/zzzeek/alembic/issues/341 was merged.
>> Fortunatelt, Mike seems to have a patch in place for Nova in order to
>> fix this https://review.openstack.org/#/c/253859/
>>
>> I'd suggest an intensive review pass on that one to make sure it's OK.
> 
> do you folks have a best practice suggestion on this?  My patch kind of
> stayed twisting in the wind for a week even though those who read it
> would have seen "hey, this is going to break on Alembic's next minor
> release!"I pinged the important people and all on it, but it still
> got no attention.

Which people were those? I guess none of us this morning knew this was
going to be an issue and were surprised that 12 hours worth of patches
had all failed.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev