On 11/30/2017 02:52 AM, Marco Varlese wrote:
Dear Ed,
On Wed, 2017-11-29 at 18:57 +0000, Ed Kern (ejk) wrote:
On Nov 29, 2017, at 3:09 AM, Marco Varlese <[email protected]
<mailto:[email protected]>> wrote:
Hi Ed,
On Wed, 2017-11-29 at 03:24 +0000, Ed Kern (ejk) wrote:
All the jobs that ive looked at vpp verify’s are still set to
‘single use slave’ So there is no re-use..
if the job is run by jenkins and is either ubuntu or centos it will
attempt to pull and install
ubuntu:
vpp-dpdk-dev
vpp-dpdk-dkms
I suppose that's exactly my point of "not-clean-state”.
Just to be clear…
Are you unhappy that it is doing those package installs before
running the make verify (or build.sh in the case of opensuse)
Or just because you think those package installs are breaking the build.
Maybe "unhappy" is a little too strong :) :) :)
I feel that being DPDK such an important piece in the VPP
infrastructure I would think it should be built during the
"VPP build process" rather than having the DPDK devel packages
installed beforehand. At the end of the day, we
even have the whole infrastructure in place to built it and link the
built DPDK in VPP so why don't we use it?
My view is that ultimately with respect to distros support of fd.io/VPP
the object is to have a dependency on an external upstream DPDK RPM.
Therefore to be able to run VPP with external DPDK RPM is critical. I
don't see the disadvantage of having DPDK RPM installed beforehand.
However, we should maintain the capability of building DPDK within the
VPP project because that is a convenience for developers and people
deploying build environments from source code with one-stop-shopping.
As I just said, this is my view and not trying to enforce it!
It will attempt to install but since it finds those packages already
installed then obviously it doesn't keep going.
well no…well I certainly should have kept going (you trimmed the log
you sent to not include the failure or a link to a specific
job.
Fair enough; my bad...
I was trying to ask and understand how is it possible that on a
"fresh booted" VM there're already packages installed.
Well when either the openstack or container image are ‘booted’
neither dpdk or vpp are installed.
32k other prereqs and base packages but (tmk with openstack clone)
not dpdk.
Sure and that's understood; I mistakenly used "fresh booted" here...
I suppose that what the message "Up-to-date DPDK package already
installed" points out, am I correct?
again i want to be careful we are on the same page..
DPDK is not installed in the base template image.
DPDK IS now installed as a package (to ubuntu and centos):
AFTER check style
BEFORE make verify or build.sh
as part of the jenkins build scripts.
Yes, indeed. We're on the same page.
+1
What I was trying to propose is not to install DPDK as a package
(ubuntu / centos) but leave everything to the Makefile/Build
infrastructure in VPP to do those steps.
That will also help us to strengthen the VPP build-infrastructure and
spot any regression when/if modifications are made to the various
Makefiles...
I don't see the necessity of this. See my comments above.
Similarly, the DPDK tarball is already downloaded when the 'curl'
command runs since the source tarball can already be found in /dpdk
So from the base template we go from no directory
template base image is spun up
git clone/fetch/etc is run
check style is run
there should be no dpdk.xxx.tar.xz anywhere at this point
make verify or build.sh (im only speaking about verify builds ) is run
as part of that ( i myself trip the download by doing make config in
the dpdk directory, never bothered to track down how it is tripped
‘normally’)
the tar.xz is pulled.
if you actually see an order different then this id be curious to see it.
thanks,
Ed
Cheers,
Marco
centos:
vpp-dpdk-devel
if running the build of opensuse or any build outside of jenkins it
will attempt to build it from scratch..
Right, I believe that should always be the approach tho...
(thats one of the issues you were seeing the other day)
https://gerrit.fd.io/r/#/c/9606/
both that dpdk-17.08 rev’d up and also the fact that they added
stable to the directory name..
and to make matters worse (only because their mirrors are hosed)
the makefile was pointed atfast.dpdk.org <http://fast.dpdk.org/>
which points to three servers that return at least two different
cksums…(So a total of 3 different a. pre 11/27 b. two post 11/27)
in my patch above i just changed it tostatic.dpdk.org
<http://static.dpdk.org/> which is slower but consistent.
Great... we could have modified my patch since yours looks pretty
similar... anyway, I abonded mine with the open yours gets merged
sooner.
Note: ignore the cpoc failures…im still bumping into oom condition
waiting on vanessa to come back and bump me up
https://rt.linuxfoundation.org/Ticket/Display.html?id=48884
On Nov 28, 2017, at 7:58 AM, Marco Varlese <[email protected]
<mailto:[email protected]>> wrote:
To add on the reported issue below, similarly, (I think that) the
DPDK tarball
can be also (always) found in the workspace hence never "freshly"
downloaded
either.
there is no tarball as part of vpp so its always getting freshly
pulled…from one location or another..
Snipped of the logs below:
---
11:00:48 @@@@ Finding source for dpdk @@@@
11:00:48 @@@@ Makefile fragment found in
/w/workspace/vpp-verify-master-
ubuntu1604/build-data/packages/dpdk.mk @@@@
11:00:48 @@@@ Source found in
/w/workspace/vpp-verify-master-ubuntu1604/dpdk
@@@@
---
On Tue, 2017-11-28 at 15:49 +0100, Marco Varlese wrote:
All,
While looking into an issue which Dave raised yesterday, I bumped
into
something
which does not look right to me.
The Jenkins jobs executing the VPP builds do not start from a
clean-state. I
would expect - for instance - not to find the DPDK package
already installed
on
the target host which builds a GERRIT patch.
well…it isn’t…its just installed post check style but before any of
the other primary build script.
I would basically like to see the
VM boot-strapping (as per the package installed by the
ci-management scripts)
and nothing else.
Anything else would be installed because of the scripts we execute to
build/install VPP (as per the Jenkins jobs). Would you
agree/disagree?
So this is how its already happening with the open stack slaves
unless something has changed.
But i will say that I disagree and its not how im doing the
container base images and thats around
make install-dep
In that every three days ill ‘bake into’ the basic build container
a run of make install-dep.
Even though it still does run make install-dep again at build time
the actual number of packages
that it has to pull in (and number of external services that have
to be available) is trimmed down
to as small a number as possible.
This is still a problem in the ‘test’ area that likes to pip
install python dependencies but are not linked
to main install-dep but i wont rant about that here...
In the Jenkins logs I can see the below all the time:
----
11:00:48 make -C dpdk install-deb
11:00:48 make[1]: Entering directory '/w/workspace/vpp-verify-master-
ubuntu1604/dpdk'
11:00:48 Building IPSec-MB 0.46 library
11:00:48 ==========================================================
11:00:48 Up-to-date DPDK package already installed
11:00:48 ==========================================================
11:00:48 make[1]: Leaving directory '/w/workspace/vpp-verify-master-
ubuntu1604/dpdk'
11:00:48
----
Also, I would like to ask if by any chance the tarball which VPP
build-system
requires (e.g. DPDK / IPSecMB / etc.) are somehow cached
somewhere (either on
the Jenkins-slave or Proxied) since I think the DPDK tarball
(currently
pulled)
may not be coming from the real location…
Wont speak for the openstack build side but I actually think it may
be worth just doing something
similar to the install-dep above with the dpdk… installing the
binaries from nexus every few days
so while it will check the nexus server for an update it normally
wouldn’t have to actually pull or install.
Will give that some thought…
and from previous thread
Since the MD5 checksum on the DPDK tarball fails; to answer your
question, no, it has never happened to me to see this specific issue
before.
I don't think there's anything specific to the openSUSE setup and/or
scripts being executed. I'd rather feel is - as I said earlier -
something to do with an hiccup on the infrastructure side. The fact
that a 'recheck' made it passing I suppose it confirms my current
theory.
well opensuse still has issues ‘passing’ or not…namely no artifacts
are ever generated/pushed
*01:43:40* [ssh-agent] Stopped.
*01:43:40* Archiving artifacts
*01:43:40* WARN: No artifacts found that match the file pattern
"build-root/*.rpm,build-root/*.deb,dpdk/*.rpm,dpdk/*.deb". Configuration error?
*01:43:40* WARN: ?build-root/*.rpm? doesn?t match anything: ?build-root? exists
but not ?build-root/*.rpm?
*01:43:40* [PostBuildScript] - Execution post build scripts.
*01:43:40* [vpp-verify-master-opensuse] $ /bin/bash
/tmp/jenkins8435214415951107237.sh
*01:43:40* Build logs: <a
href="https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-verify-master-opensuse/501";>https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-verify-master-opensuse/501</a>
*01:43:40* uname -a:
So for me opensuse ‘passing’ is something akin to a banner reading
‘Mission Accomplished’
Ed
Cheers,
--
Marco V
SUSE LINUX GmbH | GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg) Maxfeldstr. 5, D-90409, Nürnberg
_______________________________________________
vpp-dev mailing list
[email protected] <mailto:[email protected]>
https://lists.fd.io/mailman/listinfo/vpp-dev
--
Marco V
SUSE LINUX GmbH | GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg) Maxfeldstr. 5, D-90409, Nürnberg
--
Marco V
SUSE LINUX GmbH | GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg) Maxfeldstr. 5, D-90409, Nürnberg
_______________________________________________
vpp-dev mailing list
[email protected]
https://lists.fd.io/mailman/listinfo/vpp-dev
--
*Thomas F Herbert*
NFV and Fast Data Planes
Networking Group Office of the CTO
*Red Hat*
_______________________________________________
vpp-dev mailing list
[email protected]
https://lists.fd.io/mailman/listinfo/vpp-dev
