Re: [openstack-dev] [heat][telemetry] gate-ceilometer-dsvm-integration broken
On Mon, Jan 04, 2016 at 11:31:40AM +0100, Julien Danjou wrote: > On Mon, Jan 04 2016, Steven Hardy wrote: > > Hi, > > > Firstly, I'm very sorry for the breakage here, and I agree that in general > > a quick-revert is the best policy when something like this happens. > > No problem Steven, shit happens. > > It'd be even better if Heat'd move to a devstack plugin to limit this > kind of high level latency. Just by curiosity, is there a plan for that? > > > I'm a little unclear how this occurred tho, since I had a clear CI run on > > this patch: > > > > https://review.openstack.org/#/c/256315/ > > I think the dsvm tests are not the same that we run on telemetry side: > we run it with the Gnocchi backend, whereas you're likely using the > (old) Ceilometer backend. And that's the Gnocchi backend that has been > broken by the original devstack change from what I saw. > > Maybe Heat should also run this job, WDYT? > > > Given that the review latency on Devstack is quite high, it seems possible > > we'll land (2) before (1) lands, but if not then I'll re-propose it and > > hopefully figure out where I went wrong with Depends-On to confirm all is > > fixed before it lands. > > Yeah, whatever fixes the issue I'm ok with. :-) If you can fix Heat > faster, that'd be even cooler. Ok, so status update, the relevant Heat fixes have landed (thanks to ramishra and huangtianhua for investigating/fixing!): https://review.openstack.org/#/q/topic:bug/1529058 I've rechecked https://review.openstack.org/#/c/261923/1 which will hopefully show the issue is resolved and the devstack revert can be abandoned. Thanks all for your patience while we worked this issue out. Steve __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat][telemetry] gate-ceilometer-dsvm-integration broken
> > Hi, > > >> Which had a Depends-On to the devstack change, anyone know why that > >> didn't > >> fail with the CeilometerAlarmTest.test_alarm before the devstack > >> change > >> merged? > > > > It seems the test was skipped[1], as it was disabled for another > > bug[2]. > > > > [1] > > http://logs.openstack.org/15/256315/2/check/gate-heat-dsvm-functional-orig-mysql/bffccd5/console.html.gz#_2015-12-14_23_33_13_394 > > [2] https://bugs.launchpad.net/heat/+bug/1523337 > > This is unrelated, this is an old issue, this bug have already been > fixed in Aodh[1], and Heat have re-enabled the ceilometer tests [2] just > after the fix was merged. Sure, the issues are unrelated. However, the patch with ceilometer test re-enabled landed on 16th Dec[1] and CI run for Steve's patch with 'Depends-On' was on 15th Dec[2], while the test was still disabled:) Hope this clarifies why we missed the regression. [1] https://review.openstack.org/#/c/254081/ [2] https://review.openstack.org/#/c/256315/ > I think we just forget to set the status of #1523337 (heat side) when > [2] was merged. (I have just set it) > > [1] https://review.openstack.org/#/c/254078/ > [2] > http://git.openstack.org/cgit/openstack/heat/commit/?id=53e16655ab899f56bd0fd5d4997bb27a76be53df > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat][telemetry] gate-ceilometer-dsvm-integration broken
Hi, Which had a Depends-On to the devstack change, anyone know why that didn't fail with the CeilometerAlarmTest.test_alarm before the devstack change merged? It seems the test was skipped[1], as it was disabled for another bug[2]. [1] http://logs.openstack.org/15/256315/2/check/gate-heat-dsvm-functional-orig-mysql/bffccd5/console.html.gz#_2015-12-14_23_33_13_394 [2] https://bugs.launchpad.net/heat/+bug/1523337 This is unrelated, this is an old issue, this bug have already been fixed in Aodh[1], and Heat have re-enabled the ceilometer tests [2] just after the fix was merged. I think we just forget to set the status of #1523337 (heat side) when [2] was merged. (I have just set it) [1] https://review.openstack.org/#/c/254078/ [2] http://git.openstack.org/cgit/openstack/heat/commit/?id=53e16655ab899f56bd0fd5d4997bb27a76be53df __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat][telemetry] gate-ceilometer-dsvm-integration broken
On Mon, Jan 04, 2016 at 11:31:40AM +0100, Julien Danjou wrote: > On Mon, Jan 04 2016, Steven Hardy wrote: > > Hi, > > > Firstly, I'm very sorry for the breakage here, and I agree that in general > > a quick-revert is the best policy when something like this happens. > > No problem Steven, shit happens. > > It'd be even better if Heat'd move to a devstack plugin to limit this > kind of high level latency. Just by curiosity, is there a plan for that? Yes, it was discussed in a recent Heat meeting, and IIRC pas-ha was going to work on it, I agree it's a good idea and should enable more responsive updates/fixes to our devstack config. > > I'm a little unclear how this occurred tho, since I had a clear CI run on > > this patch: > > > > https://review.openstack.org/#/c/256315/ > > I think the dsvm tests are not the same that we run on telemetry side: > we run it with the Gnocchi backend, whereas you're likely using the > (old) Ceilometer backend. And that's the Gnocchi backend that has been > broken by the original devstack change from what I saw. > > Maybe Heat should also run this job, WDYT? We'll have to look into that - I was assuming all our jobs should move to using Gnocchi by default now? Rabi has acutally pointed out that we'd already skipped the heat alarm test due to a previous issue (doh!) which is probably why my Depends-On didn't catch this regression, so at least that is now understood :) Steve __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat][telemetry] gate-ceilometer-dsvm-integration broken
> On Mon, Dec 28, 2015 at 01:52:45PM +0100, Julien Danjou wrote: > > On Mon, Dec 28 2015, Rabi Mishra wrote: > > > > > Yes, this has started happening after keystone/trusts config changes by > > > the > > > devstack patch you mentioned. I've no idea how this can be fixed. As > > > Steve > > > Hardy is away, either someone with keystone knowledge should fix this or > > > we > > > merge the devstack patch revert[3] that I tested few days ago. > > > > Why don't you just revert the devstack change? > > > > This is way saner than disabling the test! Steve will be able to rework > > his initial change when he come back. > > Firstly, I'm very sorry for the breakage here, and I agree that in general > a quick-revert is the best policy when something like this happens. > > I'm a little unclear how this occurred tho, since I had a clear CI run on > this patch: > > https://review.openstack.org/#/c/256315/ > > Which had a Depends-On to the devstack change, anyone know why that didn't > fail with the CeilometerAlarmTest.test_alarm before the devstack change > merged? It seems the test was skipped[1], as it was disabled for another bug[2]. [1] http://logs.openstack.org/15/256315/2/check/gate-heat-dsvm-functional-orig-mysql/bffccd5/console.html.gz#_2015-12-14_23_33_13_394 [2] https://bugs.launchpad.net/heat/+bug/1523337 > Regardless, we've got several fixes now which can be considered: > > 1. Rabi's devstack revert: > > https://review.openstack.org/#/c/261310/ > > 2. Fix the actual issue in heat: > > https://review.openstack.org/#/q/topic:bug/1529058 > > Given that the review latency on Devstack is quite high, it seems possible > we'll land (2) before (1) lands, but if not then I'll re-propose it and > hopefully figure out where I went wrong with Depends-On to confirm all is > fixed before it lands. > > Also, there's this fix: > > https://review.openstack.org/#/c/261398/ > > I've not yet confirmed if this also fixes the issue referencing the default > domain which broke the alarm tests. > > Steve > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat][telemetry] gate-ceilometer-dsvm-integration broken
On Mon, Jan 04 2016, Steven Hardy wrote: Hi, > Firstly, I'm very sorry for the breakage here, and I agree that in general > a quick-revert is the best policy when something like this happens. No problem Steven, shit happens. It'd be even better if Heat'd move to a devstack plugin to limit this kind of high level latency. Just by curiosity, is there a plan for that? > I'm a little unclear how this occurred tho, since I had a clear CI run on > this patch: > > https://review.openstack.org/#/c/256315/ I think the dsvm tests are not the same that we run on telemetry side: we run it with the Gnocchi backend, whereas you're likely using the (old) Ceilometer backend. And that's the Gnocchi backend that has been broken by the original devstack change from what I saw. Maybe Heat should also run this job, WDYT? > Given that the review latency on Devstack is quite high, it seems possible > we'll land (2) before (1) lands, but if not then I'll re-propose it and > hopefully figure out where I went wrong with Depends-On to confirm all is > fixed before it lands. Yeah, whatever fixes the issue I'm ok with. :-) If you can fix Heat faster, that'd be even cooler. -- Julien Danjou # Free Software hacker # https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat][telemetry] gate-ceilometer-dsvm-integration broken
On Mon, Dec 28, 2015 at 01:52:45PM +0100, Julien Danjou wrote: > On Mon, Dec 28 2015, Rabi Mishra wrote: > > > Yes, this has started happening after keystone/trusts config changes by the > > devstack patch you mentioned. I've no idea how this can be fixed. As Steve > > Hardy is away, either someone with keystone knowledge should fix this or we > > merge the devstack patch revert[3] that I tested few days ago. > > Why don't you just revert the devstack change? > > This is way saner than disabling the test! Steve will be able to rework > his initial change when he come back. Firstly, I'm very sorry for the breakage here, and I agree that in general a quick-revert is the best policy when something like this happens. I'm a little unclear how this occurred tho, since I had a clear CI run on this patch: https://review.openstack.org/#/c/256315/ Which had a Depends-On to the devstack change, anyone know why that didn't fail with the CeilometerAlarmTest.test_alarm before the devstack change merged? Regardless, we've got several fixes now which can be considered: 1. Rabi's devstack revert: https://review.openstack.org/#/c/261310/ 2. Fix the actual issue in heat: https://review.openstack.org/#/q/topic:bug/1529058 Given that the review latency on Devstack is quite high, it seems possible we'll land (2) before (1) lands, but if not then I'll re-propose it and hopefully figure out where I went wrong with Depends-On to confirm all is fixed before it lands. Also, there's this fix: https://review.openstack.org/#/c/261398/ I've not yet confirmed if this also fixes the issue referencing the default domain which broke the alarm tests. Steve __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat][telemetry] gate-ceilometer-dsvm-integration broken
On Mon, Dec 28 2015, Rabi Mishra wrote: > Yes, this has started happening after keystone/trusts config changes by the > devstack patch you mentioned. I've no idea how this can be fixed. As Steve > Hardy is away, either someone with keystone knowledge should fix this or we > merge the devstack patch revert[3] that I tested few days ago. Why don't you just revert the devstack change? This is way saner than disabling the test! Steve will be able to rework his initial change when he come back. I've commented on the patches with that. Thanks guys! -- Julien Danjou /* Free Software hacker https://julien.danjou.info */ signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat][telemetry] gate-ceilometer-dsvm-integration broken
> Hi there, > > The gate for telemetry projects is broken: > > https://bugs.launchpad.net/heat/+bug/1529583 > > The failure appears in Heat from what I understand: > > BadRequest: Expecting to find domain in project - the server could not > comply with the request since it is either malformed or otherwise > incorrect. The client is assumed to be in error. (HTTP 400) > (Request-ID: req-3f39cc92-c356-4b92-9ab8-401738c8d31d Hi Julien, We're already tracking this with bug[1] for heat. As a temporary fix we've disabled the ceilometer tests from heat dsvm gate jobs[2]. Yes, this has started happening after keystone/trusts config changes by the devstack patch you mentioned. I've no idea how this can be fixed. As Steve Hardy is away, either someone with keystone knowledge should fix this or we merge the devstack patch revert[3] that I tested few days ago. [1] https://bugs.launchpad.net/heat/+bug/1529058 [2] https://review.openstack.org/#/c/261272/ [3] https://review.openstack.org/#/c/261308/ Regards, Rabi > > I've dig a bit, and I *think* that the problem lies in this recent > devsatck patch: > > https://review.openstack.org/#/c/254755/ > > Could someone from Heat tell me if I'm a good Sherlock or if I am > completely out? :) > > Cheers, > -- > Julien Danjou > // Free Software hacker > // https://julien.danjou.info > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat][telemetry] gate-ceilometer-dsvm-integration broken
Hi, Julien. I suppose, that your guess is right. Mentioned patch was merged recently and it broke our Ceilometer related functional test. There is a patch, which skip it. [1] and related bug [2] We already have a revert for this staff [3] and patch for check based on this revert [4]. [1] https://review.openstack.org/#/c/261272/ [2] https://bugs.launchpad.net/heat/+bug/1529058 [3] https://review.openstack.org/#/c/261308/ [4] https://review.openstack.org/#/c/261272/ On 28 December 2015 at 15:06, Julien Danjou wrote: > Hi there, > > The gate for telemetry projects is broken: > > https://bugs.launchpad.net/heat/+bug/1529583 > > The failure appears in Heat from what I understand: > > BadRequest: Expecting to find domain in project - the server could not > comply with the request since it is either malformed or otherwise > incorrect. The client is assumed to be in error. (HTTP 400) > (Request-ID: req-3f39cc92-c356-4b92-9ab8-401738c8d31d > > I've dig a bit, and I *think* that the problem lies in this recent > devsatck patch: > > https://review.openstack.org/#/c/254755/ > > Could someone from Heat tell me if I'm a good Sherlock or if I am > completely out? :) > > Cheers, > -- > Julien Danjou > // Free Software hacker > // https://julien.danjou.info > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > -- Regards, Sergey. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [heat][telemetry] gate-ceilometer-dsvm-integration broken
Hi there, The gate for telemetry projects is broken: https://bugs.launchpad.net/heat/+bug/1529583 The failure appears in Heat from what I understand: BadRequest: Expecting to find domain in project - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-3f39cc92-c356-4b92-9ab8-401738c8d31d I've dig a bit, and I *think* that the problem lies in this recent devsatck patch: https://review.openstack.org/#/c/254755/ Could someone from Heat tell me if I'm a good Sherlock or if I am completely out? :) Cheers, -- Julien Danjou // Free Software hacker // https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev