Re: [openstack-dev] [heat][telemetry] gate-ceilometer-dsvm-integration broken

[Rabi Mishra]

> On Mon, Dec 28, 2015 at 01:52:45PM +0100, Julien Danjou wrote:
> > On Mon, Dec 28 2015, Rabi Mishra wrote:
> > 
> > > Yes, this has started happening after keystone/trusts config changes by
> > > the
> > > devstack patch you mentioned. I've no idea how this can be fixed. As
> > > Steve
> > > Hardy is away, either someone with keystone knowledge should fix this or
> > > we
> > > merge the devstack patch revert[3] that I tested few days ago.
> > 
> > Why don't you just revert the devstack change?
> > 
> > This is way saner than disabling the test! Steve will be able to rework
> > his initial change when he come back.
> 
> Firstly, I'm very sorry for the breakage here, and I agree that in general
> a quick-revert is the best policy when something like this happens.
> 
> I'm a little unclear how this occurred tho, since I had a clear CI run on
> this patch:
> 
> https://review.openstack.org/#/c/256315/
> 
> Which had a Depends-On to the devstack change, anyone know why that didn't
> fail with the CeilometerAlarmTest.test_alarm before the devstack change
> merged?

It seems the test was skipped[1], as it was disabled for another bug[2].

[1] 
http://logs.openstack.org/15/256315/2/check/gate-heat-dsvm-functional-orig-mysql/bffccd5/console.html.gz#_2015-12-14_23_33_13_394
[2] https://bugs.launchpad.net/heat/+bug/1523337

> Regardless, we've got several fixes now which can be considered:
> 
> 1. Rabi's devstack revert:
> 
> https://review.openstack.org/#/c/261310/
> 
> 2. Fix the actual issue in heat:
> 
> https://review.openstack.org/#/q/topic:bug/1529058
> 
> Given that the review latency on Devstack is quite high, it seems possible
> we'll land (2) before (1) lands, but if not then I'll re-propose it and
> hopefully figure out where I went wrong with Depends-On to confirm all is
> fixed before it lands.
> 
> Also, there's this fix:
> 
> https://review.openstack.org/#/c/261398/
> 
> I've not yet confirmed if this also fixes the issue referencing the default
> domain which broke the alarm tests.
> 
> Steve
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat][telemetry] gate-ceilometer-dsvm-integration broken

[Steven Hardy]

On Mon, Dec 28, 2015 at 01:52:45PM +0100, Julien Danjou wrote:
> On Mon, Dec 28 2015, Rabi Mishra wrote:
> 
> > Yes, this has started happening after keystone/trusts config changes by the
> > devstack patch you mentioned. I've no idea how this can be fixed. As Steve
> > Hardy is away, either someone with keystone knowledge should fix this or we
> > merge the devstack patch revert[3] that I tested few days ago.
> 
> Why don't you just revert the devstack change?
> 
> This is way saner than disabling the test! Steve will be able to rework
> his initial change when he come back.

Firstly, I'm very sorry for the breakage here, and I agree that in general
a quick-revert is the best policy when something like this happens.

I'm a little unclear how this occurred tho, since I had a clear CI run on
this patch:

https://review.openstack.org/#/c/256315/

Which had a Depends-On to the devstack change, anyone know why that didn't
fail with the CeilometerAlarmTest.test_alarm before the devstack change
merged?

Regardless, we've got several fixes now which can be considered:

1. Rabi's devstack revert:

https://review.openstack.org/#/c/261310/

2. Fix the actual issue in heat:

https://review.openstack.org/#/q/topic:bug/1529058

Given that the review latency on Devstack is quite high, it seems possible
we'll land (2) before (1) lands, but if not then I'll re-propose it and
hopefully figure out where I went wrong with Depends-On to confirm all is
fixed before it lands.

Also, there's this fix:

https://review.openstack.org/#/c/261398/

I've not yet confirmed if this also fixes the issue referencing the default
domain which broke the alarm tests.

Steve

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat][telemetry] gate-ceilometer-dsvm-integration broken

[Rabi Mishra]

> 
> Hi,
> 
> >> Which had a Depends-On to the devstack change, anyone know why that
> >> didn't
> >> fail with the CeilometerAlarmTest.test_alarm before the devstack
> >> change
> >> merged?
> > 
> > It seems the test was skipped[1], as it was disabled for another
> > bug[2].
> > 
> > [1]
> > http://logs.openstack.org/15/256315/2/check/gate-heat-dsvm-functional-orig-mysql/bffccd5/console.html.gz#_2015-12-14_23_33_13_394
> > [2] https://bugs.launchpad.net/heat/+bug/1523337
> 
> This is unrelated, this is an old issue, this bug have already been
> fixed in Aodh[1], and Heat have re-enabled the ceilometer tests [2] just
> after the fix was merged.

Sure, the issues are unrelated. However, the patch with ceilometer test 
re-enabled landed on 16th Dec[1] and CI run for Steve's patch with 'Depends-On' 
was on 15th Dec[2], while the test was still disabled:)

Hope this clarifies why we missed the regression.

[1] https://review.openstack.org/#/c/254081/
[2] https://review.openstack.org/#/c/256315/
 
> I think we just forget to set the status of #1523337 (heat side) when
> [2] was merged. (I have just set it)
> 
> [1] https://review.openstack.org/#/c/254078/
> [2]
> http://git.openstack.org/cgit/openstack/heat/commit/?id=53e16655ab899f56bd0fd5d4997bb27a76be53df
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat][telemetry] gate-ceilometer-dsvm-integration broken

[Julien Danjou]

On Mon, Jan 04 2016, Steven Hardy wrote:

Hi,

> Firstly, I'm very sorry for the breakage here, and I agree that in general
> a quick-revert is the best policy when something like this happens.

No problem Steven, shit happens.

It'd be even better if Heat'd move to a devstack plugin to limit this
kind of high level latency. Just by curiosity, is there a plan for that?

> I'm a little unclear how this occurred tho, since I had a clear CI run on
> this patch:
>
> https://review.openstack.org/#/c/256315/

I think the dsvm tests are not the same that we run on telemetry side:
we run it with the Gnocchi backend, whereas you're likely using the
(old) Ceilometer backend. And that's the Gnocchi backend that has been
broken by the original devstack change from what I saw.

Maybe Heat should also run this job, WDYT?

> Given that the review latency on Devstack is quite high, it seems possible
> we'll land (2) before (1) lands, but if not then I'll re-propose it and
> hopefully figure out where I went wrong with Depends-On to confirm all is
> fixed before it lands.

Yeah, whatever fixes the issue I'm ok with. :-) If you can fix Heat
faster, that'd be even cooler.

-- 
Julien Danjou
# Free Software hacker
# https://julien.danjou.info


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat][telemetry] gate-ceilometer-dsvm-integration broken

[Steven Hardy]

On Mon, Jan 04, 2016 at 11:31:40AM +0100, Julien Danjou wrote:
> On Mon, Jan 04 2016, Steven Hardy wrote:
> 
> Hi,
> 
> > Firstly, I'm very sorry for the breakage here, and I agree that in general
> > a quick-revert is the best policy when something like this happens.
> 
> No problem Steven, shit happens.
> 
> It'd be even better if Heat'd move to a devstack plugin to limit this
> kind of high level latency. Just by curiosity, is there a plan for that?

Yes, it was discussed in a recent Heat meeting, and IIRC pas-ha was going
to work on it, I agree it's a good idea and should enable more responsive
updates/fixes to our devstack config.

> > I'm a little unclear how this occurred tho, since I had a clear CI run on
> > this patch:
> >
> > https://review.openstack.org/#/c/256315/
> 
> I think the dsvm tests are not the same that we run on telemetry side:
> we run it with the Gnocchi backend, whereas you're likely using the
> (old) Ceilometer backend. And that's the Gnocchi backend that has been
> broken by the original devstack change from what I saw.
> 
> Maybe Heat should also run this job, WDYT?

We'll have to look into that - I was assuming all our jobs should move to
using Gnocchi by default now?

Rabi has acutally pointed out that we'd already skipped the heat alarm test
due to a previous issue (doh!) which is probably why my Depends-On didn't
catch this regression, so at least that is now understood :)

Steve

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat][telemetry] gate-ceilometer-dsvm-integration broken

[Mehdi Abaakouk]

Hi,

Which had a Depends-On to the devstack change, anyone know why that 
didn't
fail with the CeilometerAlarmTest.test_alarm before the devstack 
change

merged?


It seems the test was skipped[1], as it was disabled for another 
bug[2].


[1]
http://logs.openstack.org/15/256315/2/check/gate-heat-dsvm-functional-orig-mysql/bffccd5/console.html.gz#_2015-12-14_23_33_13_394
[2] https://bugs.launchpad.net/heat/+bug/1523337


This is unrelated, this is an old issue, this bug have already been 
fixed in Aodh[1], and Heat have re-enabled the ceilometer tests [2] just 
after the fix was merged.


I think we just forget to set the status of #1523337 (heat side) when 
[2] was merged. (I have just set it)


[1] https://review.openstack.org/#/c/254078/
[2] 
http://git.openstack.org/cgit/openstack/heat/commit/?id=53e16655ab899f56bd0fd5d4997bb27a76be53df


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat][telemetry] gate-ceilometer-dsvm-integration broken

[Steven Hardy]

On Mon, Jan 04, 2016 at 11:31:40AM +0100, Julien Danjou wrote:
> On Mon, Jan 04 2016, Steven Hardy wrote:
> 
> Hi,
> 
> > Firstly, I'm very sorry for the breakage here, and I agree that in general
> > a quick-revert is the best policy when something like this happens.
> 
> No problem Steven, shit happens.
> 
> It'd be even better if Heat'd move to a devstack plugin to limit this
> kind of high level latency. Just by curiosity, is there a plan for that?
> 
> > I'm a little unclear how this occurred tho, since I had a clear CI run on
> > this patch:
> >
> > https://review.openstack.org/#/c/256315/
> 
> I think the dsvm tests are not the same that we run on telemetry side:
> we run it with the Gnocchi backend, whereas you're likely using the
> (old) Ceilometer backend. And that's the Gnocchi backend that has been
> broken by the original devstack change from what I saw.
> 
> Maybe Heat should also run this job, WDYT?
> 
> > Given that the review latency on Devstack is quite high, it seems possible
> > we'll land (2) before (1) lands, but if not then I'll re-propose it and
> > hopefully figure out where I went wrong with Depends-On to confirm all is
> > fixed before it lands.
> 
> Yeah, whatever fixes the issue I'm ok with. :-) If you can fix Heat
> faster, that'd be even cooler.

Ok, so status update, the relevant Heat fixes have landed (thanks to
ramishra and huangtianhua for investigating/fixing!):

https://review.openstack.org/#/q/topic:bug/1529058

I've rechecked https://review.openstack.org/#/c/261923/1 which will
hopefully show the issue is resolved and the devstack revert can be
abandoned.

Thanks all for your patience while we worked this issue out.

Steve

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat][telemetry] gate-ceilometer-dsvm-integration broken

[Sergey Kraynev]

Hi, Julien.

I suppose, that your guess is right.
Mentioned patch was merged recently and it broke our Ceilometer related
functional test.
There is a patch, which skip it. [1] and related bug [2]

We already have a revert for this staff [3] and patch for check based on
this revert [4].

[1] https://review.openstack.org/#/c/261272/
[2] https://bugs.launchpad.net/heat/+bug/1529058
[3] https://review.openstack.org/#/c/261308/
[4] https://review.openstack.org/#/c/261272/








On 28 December 2015 at 15:06, Julien Danjou  wrote:

> Hi there,
>
> The gate for telemetry projects is broken:
>
>   https://bugs.launchpad.net/heat/+bug/1529583
>
> The failure appears in Heat from what I understand:
>
>  BadRequest: Expecting to find domain in project - the server could not
>  comply with the request since it is either malformed or otherwise
>  incorrect. The client is assumed to be in error. (HTTP 400)
>  (Request-ID: req-3f39cc92-c356-4b92-9ab8-401738c8d31d
>
> I've dig a bit, and I *think* that the problem lies in this recent
> devsatck patch:
>
>   https://review.openstack.org/#/c/254755/
>
> Could someone from Heat tell me if I'm a good Sherlock or if I am
> completely out? :)
>
> Cheers,
> --
> Julien Danjou
> // Free Software hacker
> // https://julien.danjou.info
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Regards,
Sergey.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat][telemetry] gate-ceilometer-dsvm-integration broken

[Rabi Mishra]

> Hi there,
> 
> The gate for telemetry projects is broken:
> 
>   https://bugs.launchpad.net/heat/+bug/1529583
> 
> The failure appears in Heat from what I understand:
> 
>  BadRequest: Expecting to find domain in project - the server could not
>  comply with the request since it is either malformed or otherwise
>  incorrect. The client is assumed to be in error. (HTTP 400)
>  (Request-ID: req-3f39cc92-c356-4b92-9ab8-401738c8d31d

Hi Julien,

We're already tracking this with bug[1] for heat. As a temporary fix we've 
disabled the ceilometer tests from heat dsvm gate jobs[2].

Yes, this has started happening after keystone/trusts config changes by the 
devstack patch you mentioned. I've no idea how this can be fixed. As Steve 
Hardy is away, either someone with keystone knowledge should fix this or we 
merge the devstack patch revert[3] that I tested few days ago.


[1] https://bugs.launchpad.net/heat/+bug/1529058
[2] https://review.openstack.org/#/c/261272/
[3] https://review.openstack.org/#/c/261308/

Regards,
Rabi
> 
> I've dig a bit, and I *think* that the problem lies in this recent
> devsatck patch:
> 
>   https://review.openstack.org/#/c/254755/
> 
> Could someone from Heat tell me if I'm a good Sherlock or if I am
> completely out? :)
> 
> Cheers,
> --
> Julien Danjou
> // Free Software hacker
> // https://julien.danjou.info
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat][telemetry] gate-ceilometer-dsvm-integration broken

[Julien Danjou]

On Mon, Dec 28 2015, Rabi Mishra wrote:

> Yes, this has started happening after keystone/trusts config changes by the
> devstack patch you mentioned. I've no idea how this can be fixed. As Steve
> Hardy is away, either someone with keystone knowledge should fix this or we
> merge the devstack patch revert[3] that I tested few days ago.

Why don't you just revert the devstack change?

This is way saner than disabling the test! Steve will be able to rework
his initial change when he come back.

I've commented on the patches with that.

Thanks guys!

-- 
Julien Danjou
/* Free Software hacker
   https://julien.danjou.info */


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev