Re: [openstack-dev] [devstack] Bug in is_*_enabled functions?

[Dean Troyer]

On Fri, Feb 28, 2014 at 10:21 AM, Brian Haley  wrote:

> Of course now boot_from_volume.sh fails because it doesn't include
> lib/neutron,
> I've just pushed a patch for that, https://review.openstack.org/#/c/77212/
>

Thanks.  It's absence from the gate left it the poor stepchild many times.

dt

-- 

Dean Troyer
dtro...@gmail.com
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] Bug in is_*_enabled functions?

[Brian Haley]

On 02/27/2014 11:55 AM, Dean Troyer wrote:
> There is a problem in two of DevStack's exercises, floating_ips.sh and
> volume.sh, where lib/neutron is not set up properly to handle the ping_check()
> function calls.  That is what leads to what you see.
>  https://review.openstack.org/#/c/76867/ fixes the problem in the exercises.

Thanks for the patch Dean, it does fix the problem I was seeing.

Of course now boot_from_volume.sh fails because it doesn't include lib/neutron,
I've just pushed a patch for that, https://review.openstack.org/#/c/77212/

-Brian

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] Bug in is_*_enabled functions?

[Brian Haley]

On 02/27/2014 11:55 AM, Dean Troyer wrote:
> On Thu, Feb 27, 2014 at 10:34 AM, Brian Haley  > wrote:
> 
> Ok, part of this is my kernel background, where true=1 like it should be 
> :)
> So there's a -EUSERERROR there.
> 
> 
> Right.  This is Bourne/POSIX shell, forget everything logical. ;)

Carl beat that into me as well.

> That call to 'ip netns exec...' should be:
> 
> sudo /usr/local/bin/neutron-rootwrap /etc/neutron/rootwrap.conf ip netns 
> exec...
> 
> Perhaps there's something up with this test - exercises/floating_ips.sh - 
> I'll
> try running it by hand.
> 
> 
> There is a problem in two of DevStack's exercises, floating_ips.sh and
> volume.sh, where lib/neutron is not set up properly to handle the ping_check()
> function calls.  That is what leads to what you see.
>  https://review.openstack.org/#/c/76867/ fixes the problem in the exercises.
> 
> As you know, the actual issue is that Q_RR_COMMAND is not set.  Which is 
> because
> 'is_service_enabled neutron' check appears to be failing.  Note that is not 
> the
> one you found in the logs as the one in question isn't actually logged.  I've
> just re-submitted the above review with that logging turned on.
> 
> The changes to is_service_enabled() are (so far) backward-compatible in that 
> if
> is_neutron_enabled() is not declared (as is the case now) the old check in
> is_service_enabled for 'neutron' should still catch it.  That happens 
> everywhere
> else and in the DevStack test scripts, so I'm baffled and awaiting the current
> gate check on 76867.

Thanks for finding that bug and saving my sanity - yes, it all has to do with
Q_RR_COMMAND not getting setup properly.

I'll look at the patch and see if I can get things to pass when I apply it,
still trying to get that test even running correctly, which might just need a
reclone.

Thanks,

-Brian

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] Bug in is_*_enabled functions?

[Dean Troyer]

On Thu, Feb 27, 2014 at 10:34 AM, Brian Haley  wrote:

> Ok, part of this is my kernel background, where true=1 like it should be :)
> So there's a -EUSERERROR there.
>

Right.  This is Bourne/POSIX shell, forget everything logical. ;)


> That call to 'ip netns exec...' should be:
>
> sudo /usr/local/bin/neutron-rootwrap /etc/neutron/rootwrap.conf ip netns
> exec...
>
> Perhaps there's something up with this test - exercises/floating_ips.sh -
> I'll
> try running it by hand.


There is a problem in two of DevStack's exercises, floating_ips.sh and
volume.sh, where lib/neutron is not set up properly to handle the
ping_check() function calls.  That is what leads to what you see.
https://review.openstack.org/#/c/76867/ fixes the problem in the exercises.

As you know, the actual issue is that Q_RR_COMMAND is not set.  Which is
because 'is_service_enabled neutron' check appears to be failing.  Note
that is not the one you found in the logs as the one in question isn't
actually logged.  I've just re-submitted the above review with that logging
turned on.

The changes to is_service_enabled() are (so far) backward-compatible in
that if is_neutron_enabled() is not declared (as is the case now) the old
check in is_service_enabled for 'neutron' should still catch it.  That
happens everywhere else and in the DevStack test scripts, so I'm baffled
and awaiting the current gate check on 76867.

dt

-- 

Dean Troyer
dtro...@gmail.com
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] Bug in is_*_enabled functions?

[Brian Haley]

On 02/26/2014 04:23 PM, Brian Haley wrote:
> On 02/26/2014 01:36 PM, Dean Troyer wrote:
>> On Wed, Feb 26, 2014 at 11:51 AM, Brian Haley > > wrote:
>>
>> While trying to track down why Jenkins was handing out -1's in a
>> Neutron patch,
>> I was seeing errors in the devstack tests it runs.  When I dug
>> deeper it looked
>> like it wasn't properly determining that Neutron was enabled -
>> ENABLED_SERVICES
>> had multiple "q-*" entries, but 'is_service_enabled neutron' was
>> returning 0.
>>
>>
>> This is the correct return, 0 == success.

Ok, part of this is my kernel background, where true=1 like it should be :)
So there's a -EUSERERROR there.

Something is still wonky in the log below though...

> But, at least in lib/neutron, there is a check like this:
> 
> if is_service_enabled neutron; then
> ...
> fi
> 
> Which will fail with a 0 return code and miss some config.
> 
>> Can you point to a specific log example?
> 
> http://logs.openstack.org/89/70689/24/check/check-devstack-dsvm-neutron/8e137e0/console.html


> neutron.common.legacy [-] Skipping unknown group key: firewall_driver
> 2014-02-26 13:12:01.183 | ++ probe_id=ace800b2-5753-4609-b2c3-f9c87d0cc004
> 2014-02-26 13:12:01.184 | ++ echo ' ip netns exec
> qprobe-ace800b2-5753-4609-b2c3-f9c87d0cc004'
> 2014-02-26 13:12:01.184 | + probe_cmd=' ip netns exec
> qprobe-ace800b2-5753-4609-b2c3-f9c87d0cc004'
> 2014-02-26 13:12:01.185 | + [[ True = \T\r\u\e ]]
> 2014-02-26 13:12:01.185 | + check_command='while !  ip netns exec
> qprobe-ace800b2-5753-4609-b2c3-f9c87d0cc004 ping -w 1 -c 1 10.1.0.4; do sleep 
> 1;
> done'
> 2014-02-26 13:12:01.185 | + timeout 90 sh -c 'while !  ip netns exec
> qprobe-ace800b2-5753-4609-b2c3-f9c87d0cc004 ping -w 1 -c 1 10.1.0.4; do sleep 
> 1;
> done'
> 2014-02-26 13:12:01.194 | Cannot open network namespace: Permission denied
> 2014-02-26 13:12:02.201 | Cannot open network namespace: Permission denied
> 2014-02-26 13:12:03.207 | Cannot open network namespace: Permission denied
> 2014-02-26 13:12:04.213 | Cannot open network namespace: Permission denied

That call to 'ip netns exec...' should be:

sudo /usr/local/bin/neutron-rootwrap /etc/neutron/rootwrap.conf ip netns exec...

Perhaps there's something up with this test - exercises/floating_ips.sh - I'll
try running it by hand.

-Brian

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] Bug in is_*_enabled functions?

[Brian Haley]

On 02/26/2014 05:18 PM, Carl Baldwin wrote:
> Brian,
> 
> In shell it is correct to return 0 for success and non-zero for failure.

But, at least in lib/neutron, there is a check like this:

if is_service_enabled neutron; then
...
fi

Which will fail with a 0 return code and miss some config.  It seems like these
functions should return TRUE (1) in this case based on their naming scheme.

There were recent changes to these *enabled() functions a few days ago which is
part of the reason I'm asking as well, don't know if something got overlooked.

-Brian


> On Feb 26, 2014 10:54 AM, "Brian Haley"  > wrote:
> 
> While trying to track down why Jenkins was handing out -1's in a Neutron 
> patch,
> I was seeing errors in the devstack tests it runs.  When I dug deeper it 
> looked
> like it wasn't properly determining that Neutron was enabled - 
> ENABLED_SERVICES
> had multiple "q-*" entries, but 'is_service_enabled neutron' was 
> returning 0.
> 
> I boiled it down to a simple reproducer based on the many is_*_enabled()
> functions:
> 
> #!/usr/bin/env bash
> set -x
> 
> function is_foo_enabled {
> [[ ,${ENABLED_SERVICES} =~ ,"f-" ]] && return 0
> return 1
> }
> 
> ENABLED_SERVICES=f-svc
> 
> is_foo_enabled
> 
> $ ./is_foo_enabled.sh
> + ENABLED_SERVICES=f-svc
> + is_foo_enabled
> + [[ ,f-svc =~ ,f- ]]
> + return 0
> 
> So either the return values need to be swapped, or && changed to ||.  I 
> haven't
> tested is_service_enabled() but all the is_*_enabled() functions are wrong
> at least.
> 
> Is anyone else seeing this besides me?  And/or is someone already working 
> on
> fixing it?  Couldn't find a bug for it.
> 
> Thanks,
> 
> -Brian
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org 
> 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] Bug in is_*_enabled functions?

[Carl Baldwin]

Brian,

In shell it is correct to return 0 for success and non-zero for failure.

Carl
On Feb 26, 2014 10:54 AM, "Brian Haley"  wrote:

> While trying to track down why Jenkins was handing out -1's in a Neutron
> patch,
> I was seeing errors in the devstack tests it runs.  When I dug deeper it
> looked
> like it wasn't properly determining that Neutron was enabled -
> ENABLED_SERVICES
> had multiple "q-*" entries, but 'is_service_enabled neutron' was returning
> 0.
>
> I boiled it down to a simple reproducer based on the many is_*_enabled()
> functions:
>
> #!/usr/bin/env bash
> set -x
>
> function is_foo_enabled {
> [[ ,${ENABLED_SERVICES} =~ ,"f-" ]] && return 0
> return 1
> }
>
> ENABLED_SERVICES=f-svc
>
> is_foo_enabled
>
> $ ./is_foo_enabled.sh
> + ENABLED_SERVICES=f-svc
> + is_foo_enabled
> + [[ ,f-svc =~ ,f- ]]
> + return 0
>
> So either the return values need to be swapped, or && changed to ||.  I
> haven't
> tested is_service_enabled() but all the is_*_enabled() functions are wrong
> at least.
>
> Is anyone else seeing this besides me?  And/or is someone already working
> on
> fixing it?  Couldn't find a bug for it.
>
> Thanks,
>
> -Brian
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] Bug in is_*_enabled functions?

[Brian Haley]

On 02/26/2014 01:36 PM, Dean Troyer wrote:

On Wed, Feb 26, 2014 at 11:51 AM, Brian Haley mailto:brian.ha...@hp.com>> wrote:

While trying to track down why Jenkins was handing out -1's in a
Neutron patch,
I was seeing errors in the devstack tests it runs.  When I dug
deeper it looked
like it wasn't properly determining that Neutron was enabled -
ENABLED_SERVICES
had multiple "q-*" entries, but 'is_service_enabled neutron' was
returning 0.


This is the correct return, 0 == success.


But, at least in lib/neutron, there is a check like this:

if is_service_enabled neutron; then
...
fi

Which will fail with a 0 return code and miss some config.


Can you point to a specific log example?


http://logs.openstack.org/89/70689/24/check/check-devstack-dsvm-neutron/8e137e0/console.html

And snippet from log where the above call caused the test to not put a 
'sudo' at the front of something correctly:


2014-02-26 13:11:58.382 | + ping_check private 10.1.0.4 90
2014-02-26 13:11:58.382 | + is_service_enabled neutron
2014-02-26 13:11:58.385 | ++ set +o
2014-02-26 13:11:58.386 | ++ grep xtrace
2014-02-26 13:11:58.391 | + local 'xtrace=set -o xtrace'
2014-02-26 13:11:58.391 | + set +o xtrace
2014-02-26 13:11:58.392 | + return 0
2014-02-26 13:11:58.410 | + _ping_check_neutron private 10.1.0.4 90
2014-02-26 13:11:58.410 | + local from_net=private
2014-02-26 13:11:58.410 | + local ip=10.1.0.4
2014-02-26 13:11:58.410 | + local timeout_sec=90
2014-02-26 13:11:58.411 | + local expected=True
2014-02-26 13:11:58.411 | + local check_command=
2014-02-26 13:11:58.411 | ++ _get_probe_cmd_prefix private
2014-02-26 13:11:58.411 | ++ local from_net=private
2014-02-26 13:11:58.412 | +++ _get_net_id private
2014-02-26 13:11:58.412 | +++ neutron --os-tenant-name admin 
--os-username admin --os-password secret net-list

2014-02-26 13:11:58.412 | +++ grep private
2014-02-26 13:11:58.412 | +++ awk '{print $2}'
2014-02-26 13:11:59.586 | ++ net_id=779f6b6a-f477-494b-9a7a-6d81c731c4f8
2014-02-26 13:11:59.590 | +++ neutron-debug --os-tenant-name admin 
--os-username admin --os-password secret probe-list -c id -c network_id

2014-02-26 13:11:59.591 | +++ grep 779f6b6a-f477-494b-9a7a-6d81c731c4f8
2014-02-26 13:11:59.592 | +++ awk '{print $2}'
2014-02-26 13:11:59.594 | +++ head -n 1
2014-02-26 13:12:00.906 | 2014-02-26 13:12:00.876 28681 ERROR 
neutron.common.legacy [-] Skipping unknown group key: firewall_driver

2014-02-26 13:12:01.183 | ++ probe_id=ace800b2-5753-4609-b2c3-f9c87d0cc004
2014-02-26 13:12:01.184 | ++ echo ' ip netns exec 
qprobe-ace800b2-5753-4609-b2c3-f9c87d0cc004'
2014-02-26 13:12:01.184 | + probe_cmd=' ip netns exec 
qprobe-ace800b2-5753-4609-b2c3-f9c87d0cc004'

2014-02-26 13:12:01.185 | + [[ True = \T\r\u\e ]]
2014-02-26 13:12:01.185 | + check_command='while !  ip netns exec 
qprobe-ace800b2-5753-4609-b2c3-f9c87d0cc004 ping -w 1 -c 1 10.1.0.4; do 
sleep 1; done'
2014-02-26 13:12:01.185 | + timeout 90 sh -c 'while !  ip netns exec 
qprobe-ace800b2-5753-4609-b2c3-f9c87d0cc004 ping -w 1 -c 1 10.1.0.4; do 
sleep 1; done'

2014-02-26 13:12:01.194 | Cannot open network namespace: Permission denied
2014-02-26 13:12:02.201 | Cannot open network namespace: Permission denied
2014-02-26 13:12:03.207 | Cannot open network namespace: Permission denied
2014-02-26 13:12:04.213 | Cannot open network namespace: Permission denied

Thanks for any help on this.

-Brian

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] Bug in is_*_enabled functions?

[Dean Troyer]

On Wed, Feb 26, 2014 at 11:51 AM, Brian Haley  wrote:

> While trying to track down why Jenkins was handing out -1's in a Neutron
> patch,
> I was seeing errors in the devstack tests it runs.  When I dug deeper it
> looked
> like it wasn't properly determining that Neutron was enabled -
> ENABLED_SERVICES
> had multiple "q-*" entries, but 'is_service_enabled neutron' was returning
> 0.
>

This is the correct return, 0 == success.

Can you point to a specific log example?

dt

-- 

Dean Troyer
dtro...@gmail.com
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev