Re: [openstack-dev] [all] Important information for people with in-repo Zuul v3 config

[Andrea Frittoli]

On Wed, Oct 4, 2017 at 1:00 PM Jon Schlueter  wrote:

> On Wed, Oct 4, 2017 at 7:35 AM, Sean Dague  wrote:
> > On 10/04/2017 07:23 AM, Boden Russell wrote:
> >>
> >> On 10/3/17 1:37 PM, Monty Taylor wrote:
> >>> The partial rollback of Zuulv3 is in place now. Zuulv2 is acting as
> your
> >>> gate keeper once again.
> >>
> >> Since the rollback, one of the neutron-lib (v2) jobs has been
> >> consistently failing [1] with:
> >>
> >> c/_cffi_backend.c:15:17: fatal error: ffi.h: No such file or directory
> >>
> >> Is this a known issue?
> >
> > Yes, this is unrelated.
> >
> > cffi 1.11.1 was released today on pypi, without wheels. It's installed
> > super early in the job run because ansible is used to drive things.
> >
> > There is a stop gap work around that's running tests right now -
> > https://review.openstack.org/#/c/509394/1/devstack-vm-gate-wrap.sh
> >
> > If it passes I'll approve it to get folks back to work. A longer term
> > solution is probably needed.
>
> might want to consider adding --only-binary 
> to the pip install line
>
Sounds like a good safety net for the future, thanks.


>
> But it looks like cffi 1.11.1 now has wheels
>

Indeed, thank you

>
>
> Jon
>
>
> > -Sean
> >
> > --
> > Sean Dague
> > http://dague.net
> >
> >
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> --
> Jon Schlueter
> jschl...@redhat.com
> IRC: jschlueter/yazug
> Senior Software Engineer - OpenStack Productization Engineer
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Important information for people with in-repo Zuul v3 config

[Jon Schlueter]

On Wed, Oct 4, 2017 at 7:35 AM, Sean Dague  wrote:
> On 10/04/2017 07:23 AM, Boden Russell wrote:
>>
>> On 10/3/17 1:37 PM, Monty Taylor wrote:
>>> The partial rollback of Zuulv3 is in place now. Zuulv2 is acting as your
>>> gate keeper once again.
>>
>> Since the rollback, one of the neutron-lib (v2) jobs has been
>> consistently failing [1] with:
>>
>> c/_cffi_backend.c:15:17: fatal error: ffi.h: No such file or directory
>>
>> Is this a known issue?
>
> Yes, this is unrelated.
>
> cffi 1.11.1 was released today on pypi, without wheels. It's installed
> super early in the job run because ansible is used to drive things.
>
> There is a stop gap work around that's running tests right now -
> https://review.openstack.org/#/c/509394/1/devstack-vm-gate-wrap.sh
>
> If it passes I'll approve it to get folks back to work. A longer term
> solution is probably needed.

might want to consider adding --only-binary 
to the pip install line

But it looks like cffi 1.11.1 now has wheels


Jon


> -Sean
>
> --
> Sean Dague
> http://dague.net
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Jon Schlueter
jschl...@redhat.com
IRC: jschlueter/yazug
Senior Software Engineer - OpenStack Productization Engineer

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Important information for people with in-repo Zuul v3 config

[Andrea Frittoli]

On Wed, Oct 4, 2017 at 12:23 PM Boden Russell  wrote:

>
> On 10/3/17 1:37 PM, Monty Taylor wrote:
> > The partial rollback of Zuulv3 is in place now. Zuulv2 is acting as your
> > gate keeper once again.
>
> Since the rollback, one of the neutron-lib (v2) jobs has been
> consistently failing [1] with:
>
> c/_cffi_backend.c:15:17: fatal error: ffi.h: No such file or directory
>
> Is this a known issue?
>

This is cause by the latest of cffi 1.11.1 which only comes with a tarball
and no wheels as 1.11.0 used to. [0][1][2].
There is a proposed temp d-g patch up to address this [3].

[0]
https://bitbucket.org/cffi/cffi/issues/339/no-wheel-packages-for--version

[1] https://pypi.python.org/pypi/cffi/1.11.0
[2] https://pypi.python.org/pypi/cffi/1.11.1
[3] https://review.openstack.org/#/c/509394/

Andrea Frittoli (andreaf)


>
>
> [1]
>
> http://logs.openstack.org/01/508901/1/check/gate-tempest-dsvm-neutron-src-neutron-lib-ubuntu-xenial/c1d98e2/console.html#_2017-10-04_11_06_07_410466
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Important information for people with in-repo Zuul v3 config

[Rico Lin]

And we find following error in heat's zuulv2 jobs [1]:

x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall
-Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g
-fstack-protector-strong -Wformat -Werror=format-security -fPIC
-DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi
-I/usr/include/libffi -I/usr/include/python2.7 -c c/_cffi_backend.c -o
build/temp.linux-x86_64-2.7/c/_cffi_backend.o

c/_cffi_backend.c:15:17: fatal error: ffi.h: No such file or directory
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

[1]
http://logs.openstack.org/27/509227/1/check/gate-heat-dsvm-functional-orig-mysql-lbaasv2-ubuntu-xenial/68905a9/console.html#_2017-10-04_08_53_03_706042

2017-10-04 19:23 GMT+08:00 Boden Russell :

>
> On 10/3/17 1:37 PM, Monty Taylor wrote:
> > The partial rollback of Zuulv3 is in place now. Zuulv2 is acting as your
> > gate keeper once again.
>
> Since the rollback, one of the neutron-lib (v2) jobs has been
> consistently failing [1] with:
>
> c/_cffi_backend.c:15:17: fatal error: ffi.h: No such file or directory
>
> Is this a known issue?
>
>
> [1]
> http://logs.openstack.org/01/508901/1/check/gate-tempest-
> dsvm-neutron-src-neutron-lib-ubuntu-xenial/c1d98e2/console.
> html#_2017-10-04_11_06_07_410466
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
May The Force of OpenStack Be With You,

*Rico Lin*irc: ricolin
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Important information for people with in-repo Zuul v3 config

[Sean Dague]

On 10/04/2017 07:23 AM, Boden Russell wrote:
> 
> On 10/3/17 1:37 PM, Monty Taylor wrote:
>> The partial rollback of Zuulv3 is in place now. Zuulv2 is acting as your
>> gate keeper once again. 
> 
> Since the rollback, one of the neutron-lib (v2) jobs has been
> consistently failing [1] with:
> 
> c/_cffi_backend.c:15:17: fatal error: ffi.h: No such file or directory
> 
> Is this a known issue?

Yes, this is unrelated.

cffi 1.11.1 was released today on pypi, without wheels. It's installed
super early in the job run because ansible is used to drive things.

There is a stop gap work around that's running tests right now -
https://review.openstack.org/#/c/509394/1/devstack-vm-gate-wrap.sh

If it passes I'll approve it to get folks back to work. A longer term
solution is probably needed.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Important information for people with in-repo Zuul v3 config

[Boden Russell]

On 10/3/17 1:37 PM, Monty Taylor wrote:
> The partial rollback of Zuulv3 is in place now. Zuulv2 is acting as your
> gate keeper once again. 

Since the rollback, one of the neutron-lib (v2) jobs has been
consistently failing [1] with:

c/_cffi_backend.c:15:17: fatal error: ffi.h: No such file or directory

Is this a known issue?


[1]
http://logs.openstack.org/01/508901/1/check/gate-tempest-dsvm-neutron-src-neutron-lib-ubuntu-xenial/c1d98e2/console.html#_2017-10-04_11_06_07_410466

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Important information for people with in-repo Zuul v3 config

[Andrea Frittoli]

On Tue, Oct 3, 2017 at 8:37 PM Monty Taylor  wrote:

> Hi everybody,
>
> The partial rollback of Zuulv3 is in place now. Zuulv2 is acting as your
> gate keeper once again. The status page for Zuulv2 can be found at
> http://status.openstack.org/zuul and Zuulv3 can be found at
> http://zuulv3.openstack.org.
>
> With the partial rollback of v3, we've left the v3 check pipeline
> configured for everyone so that new v3 jobs can be iterated on in
> preparation for rolling forward. Doing so leaves open a potential hole
> for breakage, so ...
>
> If you propose any changes to your repos that include changes to zuul
> config files (.zuul.yaml or .zuul.d) - PLEASE make sure that Zuul v3
> runs check jobs and responds before approving the patch.
>

While work on zuulv3 continues, what's likely to happen rather often is that
patches get a +1 from Jenkins and nothing or some unrelated error from Zuul,
caused perhaps by a restart of zuulv3.

It might be handy to configure a zuulv3-only trigger for patch recheck, to
avoid
consuming the 80% share of zuulv2 nodes to re-run a couple of jobs on zuulv3
side. To that end I proposed [0], I hope you find it useful.

Andrea Frittoli (andreaf)

[0] https://review.openstack.org/#/c/509379/


>
> If you don't don't do that, you could have zuul v2 land a patch that
> contains a syntax error that would result in invalid config for v3.
> Note that this would break not only your repo - but all testing using
> Zuul v3 (in which case we would have to temporarily remove your
> repository from the v3 configuration or ask for immediate revert)!
>
> Keep in mind that as we work on diagnosing the issue that caused the
> rollback, we could be restarting v3, shutting it down for a bit or it
> could be wedged - so v3 might not respond.
>
> Make sure you get a response from v3 on any v3 related patches. Please.
>
> Thanks!
> Monty
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [all] Important information for people with in-repo Zuul v3 config

[Monty Taylor]

Hi everybody,

The partial rollback of Zuulv3 is in place now. Zuulv2 is acting as your 
gate keeper once again. The status page for Zuulv2 can be found at 
http://status.openstack.org/zuul and Zuulv3 can be found at 
http://zuulv3.openstack.org.


With the partial rollback of v3, we've left the v3 check pipeline 
configured for everyone so that new v3 jobs can be iterated on in 
preparation for rolling forward. Doing so leaves open a potential hole 
for breakage, so ...


If you propose any changes to your repos that include changes to zuul 
config files (.zuul.yaml or .zuul.d) - PLEASE make sure that Zuul v3 
runs check jobs and responds before approving the patch.


If you don't don't do that, you could have zuul v2 land a patch that 
contains a syntax error that would result in invalid config for v3.
Note that this would break not only your repo - but all testing using 
Zuul v3 (in which case we would have to temporarily remove your 
repository from the v3 configuration or ask for immediate revert)!


Keep in mind that as we work on diagnosing the issue that caused the 
rollback, we could be restarting v3, shutting it down for a bit or it 
could be wedged - so v3 might not respond.


Make sure you get a response from v3 on any v3 related patches. Please.

Thanks!
Monty

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev