Re: [openstack-dev] [Openstack-stable-maint] Stable gate status?
2014-02-20 8:57 GMT+01:00 Miguel Angel Ajo Pelayo : > I rebased the https://review.openstack.org/#/c/72576/ no-op change. And it failed in check-tempest-dsvm-neutron-pg with bug 1254890 - "Timed out waiting for thing ... to become ACTIVE" while previous check on Feb 17 failed in check-tempest-dsvm-neutron-isolated with bug 1253896 - "Attempts to verify guests are running via SSH fails. SSH connection to guest does not work." Cheers, Alan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack-stable-maint] Stable gate status?
I rebased the https://review.openstack.org/#/c/72576/ no-op change. - Original Message - > From: "Alan Pevec" > To: "openstack-stable-maint" > Cc: "OpenStack Development Mailing List" > Sent: Tuesday, February 18, 2014 7:52:23 PM > Subject: Re: [openstack-dev] [Openstack-stable-maint] Stable gate status? > > 2014-02-11 16:14 GMT+01:00 Anita Kuno: > > On 02/11/2014 04:57 AM, Alan Pevec wrote: > >> Hi Mark and Anita, > >> > >> could we declare stable/havana neutron gate jobs good enough at this > >> point? > >> There are still random failures as this no-op change shows > >> https://review.openstack.org/72576 > >> but I don't think they're stable/havana specific. > ... > > > I will reaffirm here what I had stated in IRC. > > > > If Mark McClain gives his assent for stable/havana patches to be > > approved, I will not remove Neutron stable/havana patches from the gate > > queue before they start running tests. If after they start running > > tests, they demonstrate that they are failing, I will remove them from > > the gate as a means to keep the gate flowing. If the stable/havana gate > > jobs are indeed stable, I will not be removing any patches that should > > be merged. > > As discussed on #openstack-infra last week, stable-maint team should > start looking more closely at Tempest stable/havana branch and Matthew > Treinish from Tempest core joined the stable-maint team to help us > there. > > In the meantime, we need to do something more urgently, there are > remaining failures showing up frequently in stable/havana jobs which > seem to have been fixed or at least improved on master: > > * bug 1254890 - "Timed out waiting for thing ... to become ACTIVE" > causes tempest-dsvm-* failures > resolution unclear? > > * bug 1253896 - "Attempts to verify guests are running via SSH fails. > SSH connection to guest does not work." > based on Salvatore's comment 56, I've marked it as Won't Fix in > neutron/havana and opened tempest/havana to propose what Tempest test > or jobs should skip for Havana. Please chime-in in the bug if you have > suggestions. > > > Cheers, > Alan > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack-stable-maint] Stable gate status?
2014-02-11 16:14 GMT+01:00 Anita Kuno: > On 02/11/2014 04:57 AM, Alan Pevec wrote: >> Hi Mark and Anita, >> >> could we declare stable/havana neutron gate jobs good enough at this point? >> There are still random failures as this no-op change shows >> https://review.openstack.org/72576 >> but I don't think they're stable/havana specific. ... > I will reaffirm here what I had stated in IRC. > > If Mark McClain gives his assent for stable/havana patches to be > approved, I will not remove Neutron stable/havana patches from the gate > queue before they start running tests. If after they start running > tests, they demonstrate that they are failing, I will remove them from > the gate as a means to keep the gate flowing. If the stable/havana gate > jobs are indeed stable, I will not be removing any patches that should > be merged. As discussed on #openstack-infra last week, stable-maint team should start looking more closely at Tempest stable/havana branch and Matthew Treinish from Tempest core joined the stable-maint team to help us there. In the meantime, we need to do something more urgently, there are remaining failures showing up frequently in stable/havana jobs which seem to have been fixed or at least improved on master: * bug 1254890 - "Timed out waiting for thing ... to become ACTIVE" causes tempest-dsvm-* failures resolution unclear? * bug 1253896 - "Attempts to verify guests are running via SSH fails. SSH connection to guest does not work." based on Salvatore's comment 56, I've marked it as Won't Fix in neutron/havana and opened tempest/havana to propose what Tempest test or jobs should skip for Havana. Please chime-in in the bug if you have suggestions. Cheers, Alan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack-stable-maint] Stable gate status?
On 02/11/2014 04:57 AM, Alan Pevec wrote: > Hi Mark and Anita, > > could we declare stable/havana neutron gate jobs good enough at this point? > There are still random failures as this no-op change shows > https://review.openstack.org/72576 > but I don't think they're stable/havana specific. > >>> Do we have a list of those somewhere? >> Pulled out where following Neutron patches (IMHO all innocent for gate >> breaking): >> https://review.openstack.org/62206 >> https://review.openstack.org/67214 >> https://review.openstack.org/70232 > > I've resubmitted those without "Removed from the gate..." lines in the > commit message, waiting for rechecks now. > >>> I'm particularly interested in https://review.openstack.org/#/c/66149/ as a >>> fix for >>> https://bugs.launchpad.net/keystone/+bug/1251123 > > This one is last remaining exception request for 2013.2.2 and is > waiting for the master change to be reviewed: > https://review.openstack.org/#/q/Ida39b4699ed6c568609a5121573fc3be5c4ab2f4,n,z > I hope keystone core could review this one quickly so that backport > can be updated and merged. > > Thanks! > Alan > I will reaffirm here what I had stated in IRC. If Mark McClain gives his assent for stable/havana patches to be approved, I will not remove Neutron stable/havana patches from the gate queue before they start running tests. If after they start running tests, they demonstrate that they are failing, I will remove them from the gate as a means to keep the gate flowing. If the stable/havana gate jobs are indeed stable, I will not be removing any patches that should be merged. Adding commit lines is the fastest way to submit a new patchset without affecting the code in the patch (hence removing it from the gate queue), so thank you for removing those additional lines in the commit message. Thank you, Anita. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack-stable-maint] Stable gate status?
Hi Mark and Anita, could we declare stable/havana neutron gate jobs good enough at this point? There are still random failures as this no-op change shows https://review.openstack.org/72576 but I don't think they're stable/havana specific. >> Do we have a list of those somewhere? > Pulled out where following Neutron patches (IMHO all innocent for gate > breaking): > https://review.openstack.org/62206 > https://review.openstack.org/67214 > https://review.openstack.org/70232 I've resubmitted those without "Removed from the gate..." lines in the commit message, waiting for rechecks now. >> I'm particularly interested in https://review.openstack.org/#/c/66149/ as a >> fix for >> https://bugs.launchpad.net/keystone/+bug/1251123 This one is last remaining exception request for 2013.2.2 and is waiting for the master change to be reviewed: https://review.openstack.org/#/q/Ida39b4699ed6c568609a5121573fc3be5c4ab2f4,n,z I hope keystone core could review this one quickly so that backport can be updated and merged. Thanks! Alan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack-stable-maint] Stable gate status?
On Tue, Feb 4, 2014 at 11:52 AM, Alan Pevec wrote: > 2014-02-04 Doug Hellmann : > > Do we have a list of those somewhere? > > Pulled out where following Neutron patches (IMHO all innocent for gate > breaking): > https://review.openstack.org/62206 > https://review.openstack.org/67214 > https://review.openstack.org/70232 > > > I'm particularly interested in https://review.openstack.org/#/c/66149/as a > > fix for > > https://bugs.launchpad.net/keystone/+bug/1251123 > > We can discuss it as an exception, I've opened "2013.2.2 exception > requests" thread on stable-maint last week. > I was only +1 on that patch because it's stable/havana only and I > didn't see reports from anyone running Havana with this fix, only > Kieran reported running similar patch on _Grizzly_. > There's a minor inline comment from Kieran, but that's not a blocker > afaict. > Also reviews from Keystone Core members would help. > We just discovered the issue this week at DreamHost, and the patch seems to help for havana. I noticed the inline comment, but as it's a backport I didn't think we wanted to make implementation changes? Doug > > > Cheers, > Alan > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack-stable-maint] Stable gate status?
2014-02-04 Doug Hellmann : > Do we have a list of those somewhere? Pulled out where following Neutron patches (IMHO all innocent for gate breaking): https://review.openstack.org/62206 https://review.openstack.org/67214 https://review.openstack.org/70232 > I'm particularly interested in https://review.openstack.org/#/c/66149/ as a > fix for > https://bugs.launchpad.net/keystone/+bug/1251123 We can discuss it as an exception, I've opened "2013.2.2 exception requests" thread on stable-maint last week. I was only +1 on that patch because it's stable/havana only and I didn't see reports from anyone running Havana with this fix, only Kieran reported running similar patch on _Grizzly_. There's a minor inline comment from Kieran, but that's not a blocker afaict. Also reviews from Keystone Core members would help. Cheers, Alan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack-stable-maint] Stable gate status?
On Tue, Feb 4, 2014 at 9:45 AM, Alan Pevec wrote: > 2014-02-04 Mark McClain : > > On Feb 3, 2014, at 8:27 PM, Alan Pevec wrote: > > > >> Hi Sean and Anita, > >> > >> what is your opinion about the state of stable/havana gate? > >> > >> If you think it's not stable enough to return back patches which were > >> pulled out, I'd have to announce the delay of 2013.2.2 stable release > >> until gate is fixed. > >> > >> Thanks, > >> Alan > >> > >> 2014-02-02 Alan Pevec : > >>> Hi Anita, > >>> > >>> regarding few neutron stable/havana patches you pulled out of gate e.g. > >>> > > New patchset patch-id matches previous patchset, but commit message > has changed. > > To view, visit https://review.openstack.org/70232 > >>> ... > Removed from the gate due to failing tests. Please do not reverify > or reapprove until neutron gate tests are stable. > >>> > >>> I'm not sure this is specific to neutron gate tests, there are few > >>> know (waiting for...to be become ACTIVE and SSH timeouts) since long > >>> time. > >>> I was trying to get last few patches in 2013.2.2 stable/havana release > >>> on Thursday Feb 06. > >>> Which specific gate issues are you fixing and what is ETA? > >>> > >>> Cheers, > >>> Alan > > > > We're currently working around kernel problems in the master gate and > we've seen the same bug in the stable checks too [1]. Many the current > patches up for stable merges have multiple rechecks which is very > concerning to me. It is nearly impossible to tell if these patches make > things better or worse and approving will likely cause many gate resets > annoying the entire community. We're fairly close to having an appropriate > workaround, so my vote would be to delay at least a week. > > > > mark > > > > [1] > http://logs.openstack.org/10/65810/1/check/check-tempest-dsvm-neutron-pg/2101b89/logs/kern_log.txt.gz > > > ok, since I'm handling 2013.2.2 I'm making the call: release 2013.2.2 > is now planned for Thursday Feb 13th. Freeze stays in effect, no new > proposed changes should be approved, only those few which where pulled > out. > Do we have a list of those somewhere? I'm particularly interested in https://review.openstack.org/#/c/66149/ as a fix for https://bugs.launchpad.net/keystone/+bug/1251123 Doug > > Mark, Sean, do you have a list of issues which must be fixed before we > can declare gate back to normal? > > Cheers, > Alan > > ___ > Openstack-stable-maint mailing list > openstack-stable-ma...@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-stable-maint > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack-stable-maint] Stable gate status?
2014-02-04 Mark McClain : > On Feb 3, 2014, at 8:27 PM, Alan Pevec wrote: > >> Hi Sean and Anita, >> >> what is your opinion about the state of stable/havana gate? >> >> If you think it's not stable enough to return back patches which were >> pulled out, I'd have to announce the delay of 2013.2.2 stable release >> until gate is fixed. >> >> Thanks, >> Alan >> >> 2014-02-02 Alan Pevec : >>> Hi Anita, >>> >>> regarding few neutron stable/havana patches you pulled out of gate e.g. >>> > New patchset patch-id matches previous patchset, but commit message has > changed. > To view, visit https://review.openstack.org/70232 >>> ... Removed from the gate due to failing tests. Please do not reverify or reapprove until neutron gate tests are stable. >>> >>> I'm not sure this is specific to neutron gate tests, there are few >>> know (waiting for...to be become ACTIVE and SSH timeouts) since long >>> time. >>> I was trying to get last few patches in 2013.2.2 stable/havana release >>> on Thursday Feb 06. >>> Which specific gate issues are you fixing and what is ETA? >>> >>> Cheers, >>> Alan > > We're currently working around kernel problems in the master gate and we've > seen the same bug in the stable checks too [1]. Many the current patches up > for stable merges have multiple rechecks which is very concerning to me. It > is nearly impossible to tell if these patches make things better or worse and > approving will likely cause many gate resets annoying the entire community. > We're fairly close to having an appropriate workaround, so my vote would be > to delay at least a week. > > mark > > [1] > http://logs.openstack.org/10/65810/1/check/check-tempest-dsvm-neutron-pg/2101b89/logs/kern_log.txt.gz ok, since I'm handling 2013.2.2 I'm making the call: release 2013.2.2 is now planned for Thursday Feb 13th. Freeze stays in effect, no new proposed changes should be approved, only those few which where pulled out. Mark, Sean, do you have a list of issues which must be fixed before we can declare gate back to normal? Cheers, Alan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
