Re: [vpp-dev] Verify consistently failing

[Ed Kern via Lists.Fd.Io]

I think what Ed meant to do is to speedup verify job without realizing
that test framework does a local install of vpp_papi package which is
part of the source tree.

this is correct.


So having cached virtualenv is a bad idea as
we see already.

dont agree…caching internal bits is certainly bad….I just didnt pick out in the 
rash
of 16 other packages getting installed that papi was getting compiled from the 
tree
as opposed to from local src.
The whole reason that patch and my cache exists is NOT for speed it was for
reliability. (and also not abusing dep code holders (GitHub,pypy,dpdk etc)).  I 
tracked
hundreds of failures where intermittent loss of ability to pull dep packages 
caused
entire build/verification failures.
That number has effectively dropped to zero.


I also wonder whether the caching script watches
changes in test/Makefile/PYTHON_DEPENDS and whether adding or changing
a dependency would pass a verify job.

With every verify run install-dep is run (with ubuntu16 test-dep is also run) 
this would/will
pick up any change


Ed, thoughts? I would suggest not to cache virtualenv at all.


For now im going to continue to cache but after doing so will nuke:
rm -f /var/cache/vpp/python/papi-install.done
rm -f 
/var/cache/vpp/python/virtualenv/lib/python2.7/site-packages/vpp_papi-*-py2.7.egg

the papi-install.done is nastier then I thought.  Even if ole had rev’d the 
papi package test-dep
would have not properly run do to the presence of that file.

On spot testing this will byte-compile with every test-dep on every verify run.
Ive rebuilt the cache image with these changes so we will see the results.  
(things should
already be passing, the only noticeable change will be the byte-compile of papi 
on
each and every test-dep)

Ed



Thanks,
Klement


On Tue, 2018-06-19 at 08:55 +0200, Ole Troan wrote:
Seems like my patch https://gerrit.fd.io/r/#/c/13013/
broke the verification job. I provided a fix, but for some strange
reason it seems like the verifiy build is stuck with the broken
version of the vpp_papi package.
This problem seems to persist even after I reverted 13013 with https:
//gerrit.fd.io/r/#/c/13104/

This is not reproducible locally (since make test uses the correct
python package from the build directory there).
Anyone knows how to reproduce the verify setup (or have an idea of
what's going on?)

Cheers,
Ole








-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#9648): https://lists.fd.io/g/vpp-dev/message/9648
Mute This Topic: https://lists.fd.io/mt/22427622/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Verify consistently failing

[Klement Sekera via Lists.Fd.Io]

Hi,

I created a debugging patch set https://gerrit.fd.io/r/#/c/13122/ which
hints that there is something fishy going on with the python
virtualenv.

Per https://jenkins.fd.io/job/vpp-verify-master-ubuntu1604/12067/consol
e: 

10:21:46 make -C test TEST_DIR=/w/workspace/vpp-verify-master-
ubuntu1604/test VPP_TEST_BUILD_DIR=/w/workspace/vpp-verify-master-
ubuntu1604/build-root/build-vpp-native VPP_TEST_BIN=/w/workspace/vpp-
verify-master-ubuntu1604/build-root/install-vpp-native/vpp/bin/vpp
VPP_TEST_PLUGIN_PATH=/w/workspace/vpp-verify-master-ubuntu1604/build-
root/install-vpp-native/vpp/lib/vpp_plugins:/w/workspace/vpp-verify-
master-ubuntu1604/build-root/install-vpp-native/vpp/lib64/vpp_plugins
VPP_TEST_INSTALL_PATH=/w/workspace/vpp-verify-master-ubuntu1604/build-
root/install-vpp-native/ LD_LIBRARY_PATH=/w/workspace/vpp-verify-
master-ubuntu1604/build-root/install-vpp-
native/vpp/lib/:/w/workspace/vpp-verify-master-ubuntu1604/build-
root/install-vpp-native/vpp/lib64/ EXTENDED_TESTS= PYTHON= OS_ID=ubuntu
CACHE_OUTPUT= test
10:21:46 make[2]: Entering directory '/w/workspace/vpp-verify-master-
ubuntu1604/test'
10:21:46 echo "vpp python prefix is /var/cache/vpp/python"
10:21:46 vpp python prefix is /var/cache/vpp/python

Looking at the possible causes of this in the main Makefile we get:

56ccc23f (Ed Kern   2018-04-02 16:42:48 -0600 351)
export VPP_PYTHON_PREFIX ?= $(BR)/python

commit 56ccc23fbc6244190140bd7eb57bfa75f2312c62
Author: Ed Kern 
Date:   Mon Apr 2 16:42:48 2018 -0600

Makefile: Alter VPP_PYTHON_PREFIX for preloading deps

Allow setting of VPP_PYTHON_PREFIX to alternate location
so the python prereqs can be installed into base image
Also added test-dep trigger to isolate dependency install
from actual test run

Change-Id: Ia80f5dbf71bc24eb46cd6586bcadd474ef822704
Signed-off-by: Ed Kern 

I think what Ed meant to do is to speedup verify job without realizing
that test framework does a local install of vpp_papi package which is
part of the source tree. So having cached virtualenv is a bad idea as
we see already. I also wonder whether the caching script watches
changes in test/Makefile/PYTHON_DEPENDS and whether adding or changing
a dependency would pass a verify job.

Ed, thoughts? I would suggest not to cache virtualenv at all.

Thanks,
Klement


On Tue, 2018-06-19 at 08:55 +0200, Ole Troan wrote:
> Seems like my patch https://gerrit.fd.io/r/#/c/13013/
> broke the verification job. I provided a fix, but for some strange
> reason it seems like the verifiy build is stuck with the broken
> version of the vpp_papi package.
> This problem seems to persist even after I reverted 13013 with https:
> //gerrit.fd.io/r/#/c/13104/
> 
> This is not reproducible locally (since make test uses the correct
> python package from the build directory there).
> Anyone knows how to reproduce the verify setup (or have an idea of
> what's going on?)
> 
> Cheers,
> Ole
> 
> 
> 
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#9641): https://lists.fd.io/g/vpp-dev/message/9641
Mute This Topic: https://lists.fd.io/mt/22427622/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] Verify consistently failing

[Ole Troan]

Seems like my patch https://gerrit.fd.io/r/#/c/13013/
broke the verification job. I provided a fix, but for some strange reason it 
seems like the verifiy build is stuck with the broken version of the vpp_papi 
package.
This problem seems to persist even after I reverted 13013 with 
https://gerrit.fd.io/r/#/c/13104/

This is not reproducible locally (since make test uses the correct python 
package from the build directory there).
Anyone knows how to reproduce the verify setup (or have an idea of what's going 
on?)

Cheers,
Ole


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#9639): https://lists.fd.io/g/vpp-dev/message/9639
Mute This Topic: https://lists.fd.io/mt/22427622/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-



signature.asc
Description: Message signed with OpenPGP