Re: [openstack-dev] [QA] [Ironic] [Tempest] Problem with paramiko on grenade

2016-09-09 Thread Dmitry Tantsur 

On 09/09/2016 02:08 PM, Jim Rollenhagen wrote:

On Fri, Sep 9, 2016 at 7:32 AM, Dmitry Tantsur  wrote:

On 09/08/2016 11:24 AM, Anton Arefiev wrote:


Hi folks!

We've ran into the problem with paramiko, similar to[1] in Inspector.
Our grenade job fails on TestNetworkBasicOps.test_network_basic_ops with
[2] or [3] in 90% cases.



Folks, we're close to the release, and this becomes critical. We don't
actually care much about this particular test, it's just imposed on us by
the way grenade and tempest are implemented. Is there a way to disable it,
so that we can at least release something?


We've merged a hack to ironic-inspector to prevent the offending test 
from running, so it's no longer critical. Still, I'd love to see it 
fixed properly.




Any hints are really appreciated.



Looks like reverting fix [4] solves the problem, at least grenade passed
few times with reverting patch. But it will back other problems, which
mentioned patch fixes. So we need to help with this.



Yeah, so far it seems like everything we do about this bit breaks somebody
:(


I think we should land the revert - the fix at [4] was fixing out of tree code
and problems not seen in upstream CI. With the release so close, and inspector's
gate dead in the water, I think we should revert it and find a better way to fix
the problem folks saw there.


I guess the real fix will be part of https://review.openstack.org/367478



// jim





Any suggestions are welcome.
Thanks in advance.

[1] https://bugs.launchpad.net/tempest/+bug/1615659
[2]

http://logs.openstack.org/79/358779/2/check/gate-grenade-dsvm-ironic-inspector/8b5c901/logs/grenade.sh.txt.gz#_2016-09-08_08_36_02_408
[3]

http://logs.openstack.org/95/352295/9/check/gate-grenade-dsvm-ironic-inspector/b6e3aa7/console.html#_2016-09-08_08_12_20_050120
[4] https://review.openstack.org/#/c/358610/

--
Best regards,
Anton Arefiev
Mirantis, Inc.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA] [Ironic] [Tempest] Problem with paramiko on grenade

2016-09-09 Thread Jim Rollenhagen 
On Fri, Sep 9, 2016 at 7:32 AM, Dmitry Tantsur  wrote:
> On 09/08/2016 11:24 AM, Anton Arefiev wrote:
>>
>> Hi folks!
>>
>> We've ran into the problem with paramiko, similar to[1] in Inspector.
>> Our grenade job fails on TestNetworkBasicOps.test_network_basic_ops with
>> [2] or [3] in 90% cases.
>
>
> Folks, we're close to the release, and this becomes critical. We don't
> actually care much about this particular test, it's just imposed on us by
> the way grenade and tempest are implemented. Is there a way to disable it,
> so that we can at least release something?
>
> Any hints are really appreciated.
>
>>
>> Looks like reverting fix [4] solves the problem, at least grenade passed
>> few times with reverting patch. But it will back other problems, which
>> mentioned patch fixes. So we need to help with this.
>
>
> Yeah, so far it seems like everything we do about this bit breaks somebody
> :(

I think we should land the revert - the fix at [4] was fixing out of tree code
and problems not seen in upstream CI. With the release so close, and inspector's
gate dead in the water, I think we should revert it and find a better way to fix
the problem folks saw there.

// jim

>
>>
>> Any suggestions are welcome.
>> Thanks in advance.
>>
>> [1] https://bugs.launchpad.net/tempest/+bug/1615659
>> [2]
>>
>> http://logs.openstack.org/79/358779/2/check/gate-grenade-dsvm-ironic-inspector/8b5c901/logs/grenade.sh.txt.gz#_2016-09-08_08_36_02_408
>> [3]
>>
>> http://logs.openstack.org/95/352295/9/check/gate-grenade-dsvm-ironic-inspector/b6e3aa7/console.html#_2016-09-08_08_12_20_050120
>> [4] https://review.openstack.org/#/c/358610/
>>
>> --
>> Best regards,
>> Anton Arefiev
>> Mirantis, Inc.
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA] [Ironic] [Tempest] Problem with paramiko on grenade

2016-09-09 Thread Dmitry Tantsur 

On 09/08/2016 11:24 AM, Anton Arefiev wrote:

Hi folks!

We've ran into the problem with paramiko, similar to[1] in Inspector.
Our grenade job fails on TestNetworkBasicOps.test_network_basic_ops with
[2] or [3] in 90% cases.


Folks, we're close to the release, and this becomes critical. We don't 
actually care much about this particular test, it's just imposed on us 
by the way grenade and tempest are implemented. Is there a way to 
disable it, so that we can at least release something?


Any hints are really appreciated.



Looks like reverting fix [4] solves the problem, at least grenade passed
few times with reverting patch. But it will back other problems, which
mentioned patch fixes. So we need to help with this.


Yeah, so far it seems like everything we do about this bit breaks 
somebody :(




Any suggestions are welcome.
Thanks in advance.

[1] https://bugs.launchpad.net/tempest/+bug/1615659
[2]
http://logs.openstack.org/79/358779/2/check/gate-grenade-dsvm-ironic-inspector/8b5c901/logs/grenade.sh.txt.gz#_2016-09-08_08_36_02_408
[3]
http://logs.openstack.org/95/352295/9/check/gate-grenade-dsvm-ironic-inspector/b6e3aa7/console.html#_2016-09-08_08_12_20_050120
[4] https://review.openstack.org/#/c/358610/

--
Best regards,
Anton Arefiev
Mirantis, Inc.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev