[dpdk-dev] [PATCH] validate_abi: build faster by augmenting make with job count

[Wiles, Keith]

>> Neil
>> 
> Sorry for the delayed response, I've been on vacation.
> 
>> Your modified copy of make has no bearing on the topic we are taking about 
>> customers using dpdk in standard distros right?
>> 
> !?  I really don't know what you're saying here.  My only reason for 
> commenting
> on my copy of make was to consider an explination for why make -j 0 might be
> working for me, and not for you.  I've since uninstalled and resinalled make 
> on
> my system, and my copy now behaves as yours does
> 
> But thats all really irrelevant.  I don't know what you mean by this not 
> having
> any bearing on the conversation since we're talking about customers using dpdk
> in a distro.  We're not really talking about that at all (if we were using 
> make
> would be a moot point, since distro users tend to only use pre-compiled
> binaries).  Were talking about accelerating the build process when comparing
> ABIs on two different dpdk versions, thats all.

Neil, (I am trying to not read your style of text as condescending and I will 
try to not do that as well)

Not everyone uses DPDK from prebuilt libraries and we need to support them as 
well, correct?

> 
>> Seems odd to me to send this out with 0 or lspci as it may fail because of 
>> no lspci and will fail on all Ubuntu systems. 
>> 
> Again, don't really understand what your saying.  If you look at the patch,
> neither of your assertions are true.  With this patch and no other change, the
> validate_abi script behaves exactly as it does now.  The only thing I've done 
> is
> add a check for the DPDK_MAKE_JOBS environment variable, and if its not set,
> either:
> 
> a) Set DPDK_MAKE_JOBS to 1 if lscpu doesn't exist on the system
> b) Set DPDK_MAKE_JOBS to the number of online cpus if lscpu does exist
> 
> All of that gets overridden if you manually set DPDK_MAKE_JOBS to something
> else.
> 
> seems pretty straightforward to me.

At this point do we need to add yet another environment variable to get the 
correct behavior with DPDK. DPDK is very simple to build today and I worry we 
keep adding special variables to build DPDK. Can we just use a cleaner default, 
then adding more and more special build requirements? Adding this one is fine, 
but it also means the customer must read the docs to find this new variable.

DPDK should be build able with the least amount of docs to read, then they can 
read the docs more later. Just looking at how the developer can get started 
building DPDK without RTFM problem. At some point they need to read the docs to 
possibly runs the examples, but to build DPDK should very simple IMO.

> 
>> If we ship with 1 then why even bother the adding code and if I have to edit 
>> the file or some other method to get better compile performance then why 
>> bother as well.
>> 
> Please stop and think for a minute.  Why would you need to edit a file to 
> change
> anything?  If lscpu exists, then everything happens automatically.  If it
> doesn't you can still just run:
> export DPDK_MAKE_JOBS=$BIGNUMBER; validate_abi.sh 

Please do not add extra environment variable we would start to get to the point 
of having so many pre-build requirements it becomes the private knowledge to 
just build DPDK or a huge setup/RTFM problem.

> 
> and it works fine.  If ubuntu has some other utiilty besides lscpu to parse 
> cpu
> counts, fine, we can add that in as a feature, but I don't see why that should
> stop other, non-ubuntu systems from taking advantage of this.
> 
>> Setting the value to some large number does not make any sense to me and if 
>> I have to edit file every time or maintain a patch just seems silly. 
>> 
> Goodness Keith, stop for just a minute with the editing the file train your 
> on.
> Its an environment variable, you don't have to edit a file to change it.

Yes Neil, you also need to stop an think about what you are placing on the 
developer to build DPDK. This one little problem is not the real issue to me, 
but a symptom of a growing problem in DPDK around how DPDK is build and the 
amount of knowledge or setup it requires to do this one simple task.

> 
>> It just seems easier to set it to -j and not use lspci at all. This way we 
>> all win as I am not following your logic at all.
>> 
> This doesn't even make any sense.  If you set it to -j then you get make 
> issuing
> jobs at the max rate make can issue them, which is potentially fine, but may 
> not
> be what developers want in the event they don't want this script to monopolize
> their system.  The way its written now lets people get reasonable behavior in
> most cases, and opt-in to the extreeme case should they desire.  That makes 
> far
> more sense to me, then just chewing up as much cpu as possible all the time.

I only suggest -j as this would give the developer the best build performance 
without having to require lscpu or setting up yet another build environment 
variable. The lscpu command does not exist on all systems today and other 
non-Linux based 

[dpdk-dev] [PATCH] validate_abi: build faster by augmenting make with job count

[Neil Horman]

On Mon, Aug 01, 2016 at 04:16:12PM +, Wiles, Keith wrote:
> 
> >> Neil
> >> 
> > Sorry for the delayed response, I've been on vacation.
> > 
> >> Your modified copy of make has no bearing on the topic we are taking about 
> >> customers using dpdk in standard distros right?
> >> 
> > !?  I really don't know what you're saying here.  My only reason for 
> > commenting
> > on my copy of make was to consider an explination for why make -j 0 might be
> > working for me, and not for you.  I've since uninstalled and resinalled 
> > make on
> > my system, and my copy now behaves as yours does
> > 
> > But thats all really irrelevant.  I don't know what you mean by this not 
> > having
> > any bearing on the conversation since we're talking about customers using 
> > dpdk
> > in a distro.  We're not really talking about that at all (if we were using 
> > make
> > would be a moot point, since distro users tend to only use pre-compiled
> > binaries).  Were talking about accelerating the build process when comparing
> > ABIs on two different dpdk versions, thats all.
> 
> Neil, (I am trying to not read your style of text as condescending and I will 
> try to not do that as well)
> 
> Not everyone uses DPDK from prebuilt libraries and we need to support them as 
> well, correct?
> 
Correct, which is why I didn't understand your initial comment:
"Your modified copy of make has no bearing on the topic we are taking about
customers using dpdk in standard distros right?"

I read that as you saying that the topic we are discussing here is DPDK use in
standard distros, to which I am replying "No, that is not what we are talking
about here at all"

Standard Distros, as I am involved with them, are standard because the end user
typically strives to never build software included in the distro themselves.  As
such, this patch has no bearing whatsoever on those end users, because they
expect to just use a pre-built binary that conforms to a given ABI level.  They
have no need for this code

As you note, of course other upstream developers don't use pre-built binaries,
and that is who this change is targeting

> > 
> >> Seems odd to me to send this out with 0 or lspci as it may fail because of 
> >> no lspci and will fail on all Ubuntu systems. 
> >> 
> > Again, don't really understand what your saying.  If you look at the patch,
> > neither of your assertions are true.  With this patch and no other change, 
> > the
> > validate_abi script behaves exactly as it does now.  The only thing I've 
> > done is
> > add a check for the DPDK_MAKE_JOBS environment variable, and if its not set,
> > either:
> > 
> > a) Set DPDK_MAKE_JOBS to 1 if lscpu doesn't exist on the system
> > b) Set DPDK_MAKE_JOBS to the number of online cpus if lscpu does exist
> > 
> > All of that gets overridden if you manually set DPDK_MAKE_JOBS to something
> > else.
> > 
> > seems pretty straightforward to me.
> 
> At this point do we need to add yet another environment variable to get the 
> correct behavior with DPDK. DPDK is very simple to build today and I worry we 
> keep adding special variables to build DPDK. Can we just use a cleaner 
> default, then adding more and more special build requirements? Adding this 
> one is fine, but it also means the customer must read the docs to find this 
> new variable.
> 
Please read back through the thread.  The DPDK_MAKE_JOBS environment variable
was specifically used because it already exists in the build (see
test-build.sh).  Thomas specifically asked me to change the environment variable
name so that it can be resued.  We're not adding anything here that isn't
already there in other locations.

> DPDK should be build able with the least amount of docs to read, then they 
> can read the docs more later. Just looking at how the developer can get 
> started building DPDK without RTFM problem. At some point they need to read 
> the docs to possibly runs the examples, but to build DPDK should very simple 
> IMO.
So, this script has nothing to do with actually building the DPDK, only
analyzing differences in ABI between two arbitrary levels.  Nothing about this
change makes building the DPDK harder, easier, or in any way different.

> 
> > 
> >> If we ship with 1 then why even bother the adding code and if I have to 
> >> edit the file or some other method to get better compile performance then 
> >> why bother as well.
> >> 
> > Please stop and think for a minute.  Why would you need to edit a file to 
> > change
> > anything?  If lscpu exists, then everything happens automatically.  If it
> > doesn't you can still just run:
> > export DPDK_MAKE_JOBS=$BIGNUMBER; validate_abi.sh 
> 
> Please do not add extra environment variable we would start to get to the 
> point of having so many pre-build requirements it becomes the private 
> knowledge to just build DPDK or a huge setup/RTFM problem.
> 
See above, I'm getting the impression that you're just arguing without actually
looking at the code first.

> > 
> > and it 

[dpdk-dev] [PATCH] validate_abi: build faster by augmenting make with job count

[Neil Horman]

On Sun, Jul 24, 2016 at 06:08:00PM +, Wiles, Keith wrote:
> 
> 
> Sent from my iPhone
> 
> > On Jul 21, 2016, at 1:34 PM, Neil Horman  wrote:
> > 
> >> On Thu, Jul 21, 2016 at 03:22:45PM +, Wiles, Keith wrote:
> >> 
> >>> On Jul 21, 2016, at 10:06 AM, Neil Horman  wrote:
> >>> 
> >>> On Thu, Jul 21, 2016 at 02:09:19PM +, Wiles, Keith wrote:
>  
> > On Jul 21, 2016, at 8:54 AM, Neil Horman  
> > wrote:
> > 
> > On Wed, Jul 20, 2016 at 10:32:28PM +, Wiles, Keith wrote:
> >> 
> >>> On Jul 20, 2016, at 3:16 PM, Neil Horman  
> >>> wrote:
> >>> 
> >>> On Wed, Jul 20, 2016 at 07:47:32PM +, Wiles, Keith wrote:
>  
> > On Jul 20, 2016, at 12:48 PM, Neil Horman  
> > wrote:
> > 
> > On Wed, Jul 20, 2016 at 07:40:49PM +0200, Thomas Monjalon wrote:
> >> 2016-07-20 13:09, Neil Horman:
> >>> From: Neil Horman 
> >>> 
> >>> John Mcnamara and I were discussing enhacing the validate_abi 
> >>> script to build
> >>> the dpdk tree faster with multiple jobs.  Theres no reason not to 
> >>> do it, so this
> >>> implements that requirement.  It uses a MAKE_JOBS variable that 
> >>> can be set by
> >>> the user to limit the job count.  By default the job count is set 
> >>> to the number
> >>> of online cpus.
> >> 
> >> Please could you use the variable name DPDK_MAKE_JOBS?
> >> This name is already used in scripts/test-build.sh.
> > Sure
> > 
> >>> +if [ -z "$MAKE_JOBS" ]
> >>> +then
> >>> +# This counts the number of cpus on the system
> >>> +MAKE_JOBS=`lscpu -p=cpu | grep -v "#" | wc -l`
> >>> +fi
> >> 
> >> Is lscpu common enough?
> > I'm not sure how to answer that.  lscpu is part of the util-linux 
> > package, which
> > is part of any base install.  Theres a variant for BSD, but I'm not 
> > sure how
> > common it is there.
> > Neil
> > 
> >> Another acceptable default would be just "-j" without any number.
> >> It would make the number of jobs unlimited.
>  
>  I think the best is just use -j as it tries to use the correct 
>  number of jobs based on the number of cores, right?
> >>> -j with no argument (or -j 0), is sort of, maybe what you want.  With 
> >>> either of
> >>> those options, make will just issue jobs as fast as it processes 
> >>> dependencies.
> >>> Dependent on how parallel the build is, that can lead to tons of 
> >>> waiting process
> >>> (i.e. more than your number of online cpus), which can actually hurt 
> >>> your build
> >>> time.
> >> 
> >> I read the manual and looked at the code, which supports your 
> >> statement. (I think I had some statement on stack overflow and the 
> >> last time I believe anything on the internet :-) I have not seen a lot 
> >> of differences in compile times with -j on my system. Mostly I suspect 
> >> it is the number of paths in the dependency, cores and memory on the 
> >> system.
> >> 
> >> I have 72 lcores or 2 sockets, 18 cores per socket. Xeon 2.3Ghz cores.
> >> 
> >> $ export RTE_TARGET=x86_64-native-linuxapp-gcc 
> >> 
> >> $ time make install T=${RTE_TARGET}
> >> real0m59.445s user0m27.344s sys0m7.040s
> >> 
> >> $ time make install T=${RTE_TARGET} -j
> >> real0m26.584s user0m14.380s sys0m5.120s
> >> 
> >> # Remove the x86_64-native-linuxapp-gcc
> >> 
> >> $ time make install T=${RTE_TARGET} -j 72
> >> real0m23.454s user0m10.832s sys0m4.664s
> >> 
> >> $ time make install T=${RTE_TARGET} -j 8
> >> real0m23.812s user0m10.672s sys0m4.276s
> >> 
> >> cd x86_64-native-linuxapp-gcc
> >> $ make clean
> >> $ time make
> >> real0m28.539s user0m9.820s sys0m3.620s
> >> 
> >> # Do a make clean between each build.
> >> 
> >> $ time make -j
> >> real0m7.217s user0m6.532s sys0m2.332s
> >> 
> >> $ time make -j 8
> >> real0m8.256s user0m6.472s sys0m2.456s
> >> 
> >> $ time make -j 72
> >> real0m6.866s user0m6.184s sys0m2.216s
> >> 
> >> Just the real time numbers in the following table.
> >> 
> >> processes real Time   depdirs
> >>   no -j 59.4sYes
> >> -j 8 23.8sYes
> >>-j 7223.5sYes
> >>  -j   26.5sYes
> >> 
> >>   no -j 28.5s No
> >> -j 8   8.2s No
> >>-j 72  6.8s No
> >>  -j 7.2s No
> >> 
> >> Looks like the depdirs build time on my system:
> 

[dpdk-dev] [PATCH] validate_abi: build faster by augmenting make with job count

[Wiles, Keith]

Sent from my iPhone

> On Jul 21, 2016, at 1:34 PM, Neil Horman  wrote:
> 
>> On Thu, Jul 21, 2016 at 03:22:45PM +, Wiles, Keith wrote:
>> 
>>> On Jul 21, 2016, at 10:06 AM, Neil Horman  wrote:
>>> 
>>> On Thu, Jul 21, 2016 at 02:09:19PM +, Wiles, Keith wrote:
 
> On Jul 21, 2016, at 8:54 AM, Neil Horman  wrote:
> 
> On Wed, Jul 20, 2016 at 10:32:28PM +, Wiles, Keith wrote:
>> 
>>> On Jul 20, 2016, at 3:16 PM, Neil Horman  
>>> wrote:
>>> 
>>> On Wed, Jul 20, 2016 at 07:47:32PM +, Wiles, Keith wrote:
 
> On Jul 20, 2016, at 12:48 PM, Neil Horman  
> wrote:
> 
> On Wed, Jul 20, 2016 at 07:40:49PM +0200, Thomas Monjalon wrote:
>> 2016-07-20 13:09, Neil Horman:
>>> From: Neil Horman 
>>> 
>>> John Mcnamara and I were discussing enhacing the validate_abi 
>>> script to build
>>> the dpdk tree faster with multiple jobs.  Theres no reason not to 
>>> do it, so this
>>> implements that requirement.  It uses a MAKE_JOBS variable that can 
>>> be set by
>>> the user to limit the job count.  By default the job count is set 
>>> to the number
>>> of online cpus.
>> 
>> Please could you use the variable name DPDK_MAKE_JOBS?
>> This name is already used in scripts/test-build.sh.
> Sure
> 
>>> +if [ -z "$MAKE_JOBS" ]
>>> +then
>>> +# This counts the number of cpus on the system
>>> +MAKE_JOBS=`lscpu -p=cpu | grep -v "#" | wc -l`
>>> +fi
>> 
>> Is lscpu common enough?
> I'm not sure how to answer that.  lscpu is part of the util-linux 
> package, which
> is part of any base install.  Theres a variant for BSD, but I'm not 
> sure how
> common it is there.
> Neil
> 
>> Another acceptable default would be just "-j" without any number.
>> It would make the number of jobs unlimited.
 
 I think the best is just use -j as it tries to use the correct number 
 of jobs based on the number of cores, right?
>>> -j with no argument (or -j 0), is sort of, maybe what you want.  With 
>>> either of
>>> those options, make will just issue jobs as fast as it processes 
>>> dependencies.
>>> Dependent on how parallel the build is, that can lead to tons of 
>>> waiting process
>>> (i.e. more than your number of online cpus), which can actually hurt 
>>> your build
>>> time.
>> 
>> I read the manual and looked at the code, which supports your statement. 
>> (I think I had some statement on stack overflow and the last time I 
>> believe anything on the internet :-) I have not seen a lot of 
>> differences in compile times with -j on my system. Mostly I suspect it 
>> is the number of paths in the dependency, cores and memory on the system.
>> 
>> I have 72 lcores or 2 sockets, 18 cores per socket. Xeon 2.3Ghz cores.
>> 
>> $ export RTE_TARGET=x86_64-native-linuxapp-gcc 
>> 
>> $ time make install T=${RTE_TARGET}
>> real0m59.445s user0m27.344s sys0m7.040s
>> 
>> $ time make install T=${RTE_TARGET} -j
>> real0m26.584s user0m14.380s sys0m5.120s
>> 
>> # Remove the x86_64-native-linuxapp-gcc
>> 
>> $ time make install T=${RTE_TARGET} -j 72
>> real0m23.454s user0m10.832s sys0m4.664s
>> 
>> $ time make install T=${RTE_TARGET} -j 8
>> real0m23.812s user0m10.672s sys0m4.276s
>> 
>> cd x86_64-native-linuxapp-gcc
>> $ make clean
>> $ time make
>> real0m28.539s user0m9.820s sys0m3.620s
>> 
>> # Do a make clean between each build.
>> 
>> $ time make -j
>> real0m7.217s user0m6.532s sys0m2.332s
>> 
>> $ time make -j 8
>> real0m8.256s user0m6.472s sys0m2.456s
>> 
>> $ time make -j 72
>> real0m6.866s user0m6.184s sys0m2.216s
>> 
>> Just the real time numbers in the following table.
>> 
>> processes real Time   depdirs
>>   no -j 59.4sYes
>> -j 8 23.8sYes
>>-j 7223.5sYes
>>  -j   26.5sYes
>> 
>>   no -j 28.5s No
>> -j 8   8.2s No
>>-j 72  6.8s No
>>  -j 7.2s No
>> 
>> Looks like the depdirs build time on my system:
>> $ make clean -j
>> $ rm .depdirs
>> $ time make -j
>> real0m23.734s user0m11.228s sys0m4.844s
>> 
>> About 16 seconds, which is not a lot of savings. Now the difference from 
>> no -j to -j is a lot, but the difference between -j and -j  
>> is not a huge saving. This leads me back to 

[dpdk-dev] [PATCH] validate_abi: build faster by augmenting make with job count

[Wiles, Keith]

> On Jul 21, 2016, at 10:06 AM, Neil Horman  wrote:
> 
> On Thu, Jul 21, 2016 at 02:09:19PM +, Wiles, Keith wrote:
>> 
>>> On Jul 21, 2016, at 8:54 AM, Neil Horman  wrote:
>>> 
>>> On Wed, Jul 20, 2016 at 10:32:28PM +, Wiles, Keith wrote:
 
> On Jul 20, 2016, at 3:16 PM, Neil Horman  wrote:
> 
> On Wed, Jul 20, 2016 at 07:47:32PM +, Wiles, Keith wrote:
>> 
>>> On Jul 20, 2016, at 12:48 PM, Neil Horman  wrote:
>>> 
>>> On Wed, Jul 20, 2016 at 07:40:49PM +0200, Thomas Monjalon wrote:
 2016-07-20 13:09, Neil Horman:
> From: Neil Horman 
> 
> John Mcnamara and I were discussing enhacing the validate_abi script 
> to build
> the dpdk tree faster with multiple jobs.  Theres no reason not to do 
> it, so this
> implements that requirement.  It uses a MAKE_JOBS variable that can 
> be set by
> the user to limit the job count.  By default the job count is set to 
> the number
> of online cpus.
 
 Please could you use the variable name DPDK_MAKE_JOBS?
 This name is already used in scripts/test-build.sh.
 
>>> Sure
>>> 
> +if [ -z "$MAKE_JOBS" ]
> +then
> + # This counts the number of cpus on the system
> + MAKE_JOBS=`lscpu -p=cpu | grep -v "#" | wc -l`
> +fi
 
 Is lscpu common enough?
 
>>> I'm not sure how to answer that.  lscpu is part of the util-linux 
>>> package, which
>>> is part of any base install.  Theres a variant for BSD, but I'm not 
>>> sure how
>>> common it is there.
>>> Neil
>>> 
 Another acceptable default would be just "-j" without any number.
 It would make the number of jobs unlimited.
>> 
>> I think the best is just use -j as it tries to use the correct number of 
>> jobs based on the number of cores, right?
>> 
> -j with no argument (or -j 0), is sort of, maybe what you want.  With 
> either of
> those options, make will just issue jobs as fast as it processes 
> dependencies.
> Dependent on how parallel the build is, that can lead to tons of waiting 
> process
> (i.e. more than your number of online cpus), which can actually hurt your 
> build
> time.
 
 I read the manual and looked at the code, which supports your statement. 
 (I think I had some statement on stack overflow and the last time I 
 believe anything on the internet :-) I have not seen a lot of differences 
 in compile times with -j on my system. Mostly I suspect it is the number 
 of paths in the dependency, cores and memory on the system.
 
 I have 72 lcores or 2 sockets, 18 cores per socket. Xeon 2.3Ghz cores.
 
 $ export RTE_TARGET=x86_64-native-linuxapp-gcc 
 
 $ time make install T=${RTE_TARGET}
 real   0m59.445s user  0m27.344s sys   0m7.040s
 
 $ time make install T=${RTE_TARGET} -j
 real   0m26.584s user  0m14.380s sys   0m5.120s
 
 # Remove the x86_64-native-linuxapp-gcc
 
 $ time make install T=${RTE_TARGET} -j 72
 real   0m23.454s user  0m10.832s sys   0m4.664s
 
 $ time make install T=${RTE_TARGET} -j 8
 real   0m23.812s user  0m10.672s sys   0m4.276s
 
 cd x86_64-native-linuxapp-gcc
 $ make clean
 $ time make
 real   0m28.539s user  0m9.820s sys0m3.620s
 
 # Do a make clean between each build.
 
 $ time make -j
 real   0m7.217s user   0m6.532s sys0m2.332s
 
 $ time make -j 8
 real   0m8.256s user   0m6.472s sys0m2.456s
 
 $ time make -j 72
 real   0m6.866s user   0m6.184s sys0m2.216s
 
 Just the real time numbers in the following table.
 
 processes real Time   depdirs
no -j 59.4sYes
  -j 8 23.8sYes
 -j 7223.5sYes
   -j   26.5sYes
 
no -j 28.5s No
  -j 8   8.2s No
 -j 72  6.8s No
   -j 7.2s No
 
 Looks like the depdirs build time on my system:
 $ make clean -j
 $ rm .depdirs
 $ time make -j
 real   0m23.734s user  0m11.228s sys   0m4.844s
 
 About 16 seconds, which is not a lot of savings. Now the difference from 
 no -j to -j is a lot, but the difference between -j and -j  is 
 not a huge saving. This leads me back to over engineering the problem when 
 ?-j? would work just as well here.
 
 Even on my MacBook Pro i7 system the difference is not that much 1m8s 
 without depdirs build for -j in a VirtualBox with all 4 cores 8G RAM. 
 Compared to 1m13s with -j 4 option.
 
 I just wonder if it makes a lot of sense to use cpuinfo in this given case 
 

[dpdk-dev] [PATCH] validate_abi: build faster by augmenting make with job count

[Neil Horman]

On Thu, Jul 21, 2016 at 03:22:45PM +, Wiles, Keith wrote:
> 
> > On Jul 21, 2016, at 10:06 AM, Neil Horman  wrote:
> > 
> > On Thu, Jul 21, 2016 at 02:09:19PM +, Wiles, Keith wrote:
> >> 
> >>> On Jul 21, 2016, at 8:54 AM, Neil Horman  wrote:
> >>> 
> >>> On Wed, Jul 20, 2016 at 10:32:28PM +, Wiles, Keith wrote:
>  
> > On Jul 20, 2016, at 3:16 PM, Neil Horman  
> > wrote:
> > 
> > On Wed, Jul 20, 2016 at 07:47:32PM +, Wiles, Keith wrote:
> >> 
> >>> On Jul 20, 2016, at 12:48 PM, Neil Horman  
> >>> wrote:
> >>> 
> >>> On Wed, Jul 20, 2016 at 07:40:49PM +0200, Thomas Monjalon wrote:
>  2016-07-20 13:09, Neil Horman:
> > From: Neil Horman 
> > 
> > John Mcnamara and I were discussing enhacing the validate_abi 
> > script to build
> > the dpdk tree faster with multiple jobs.  Theres no reason not to 
> > do it, so this
> > implements that requirement.  It uses a MAKE_JOBS variable that can 
> > be set by
> > the user to limit the job count.  By default the job count is set 
> > to the number
> > of online cpus.
>  
>  Please could you use the variable name DPDK_MAKE_JOBS?
>  This name is already used in scripts/test-build.sh.
>  
> >>> Sure
> >>> 
> > +if [ -z "$MAKE_JOBS" ]
> > +then
> > +   # This counts the number of cpus on the system
> > +   MAKE_JOBS=`lscpu -p=cpu | grep -v "#" | wc -l`
> > +fi
>  
>  Is lscpu common enough?
>  
> >>> I'm not sure how to answer that.  lscpu is part of the util-linux 
> >>> package, which
> >>> is part of any base install.  Theres a variant for BSD, but I'm not 
> >>> sure how
> >>> common it is there.
> >>> Neil
> >>> 
>  Another acceptable default would be just "-j" without any number.
>  It would make the number of jobs unlimited.
> >> 
> >> I think the best is just use -j as it tries to use the correct number 
> >> of jobs based on the number of cores, right?
> >> 
> > -j with no argument (or -j 0), is sort of, maybe what you want.  With 
> > either of
> > those options, make will just issue jobs as fast as it processes 
> > dependencies.
> > Dependent on how parallel the build is, that can lead to tons of 
> > waiting process
> > (i.e. more than your number of online cpus), which can actually hurt 
> > your build
> > time.
>  
>  I read the manual and looked at the code, which supports your statement. 
>  (I think I had some statement on stack overflow and the last time I 
>  believe anything on the internet :-) I have not seen a lot of 
>  differences in compile times with -j on my system. Mostly I suspect it 
>  is the number of paths in the dependency, cores and memory on the system.
>  
>  I have 72 lcores or 2 sockets, 18 cores per socket. Xeon 2.3Ghz cores.
>  
>  $ export RTE_TARGET=x86_64-native-linuxapp-gcc 
>  
>  $ time make install T=${RTE_TARGET}
>  real 0m59.445s user  0m27.344s sys   0m7.040s
>  
>  $ time make install T=${RTE_TARGET} -j
>  real 0m26.584s user  0m14.380s sys   0m5.120s
>  
>  # Remove the x86_64-native-linuxapp-gcc
>  
>  $ time make install T=${RTE_TARGET} -j 72
>  real 0m23.454s user  0m10.832s sys   0m4.664s
>  
>  $ time make install T=${RTE_TARGET} -j 8
>  real 0m23.812s user  0m10.672s sys   0m4.276s
>  
>  cd x86_64-native-linuxapp-gcc
>  $ make clean
>  $ time make
>  real 0m28.539s user  0m9.820s sys0m3.620s
>  
>  # Do a make clean between each build.
>  
>  $ time make -j
>  real 0m7.217s user   0m6.532s sys0m2.332s
>  
>  $ time make -j 8
>  real 0m8.256s user   0m6.472s sys0m2.456s
>  
>  $ time make -j 72
>  real 0m6.866s user   0m6.184s sys0m2.216s
>  
>  Just the real time numbers in the following table.
>  
>  processes real Time   depdirs
> no -j 59.4sYes
>   -j 8 23.8sYes
>  -j 7223.5sYes
>    -j   26.5sYes
>  
> no -j 28.5s No
>   -j 8   8.2s No
>  -j 72  6.8s No
>    -j 7.2s No
>  
>  Looks like the depdirs build time on my system:
>  $ make clean -j
>  $ rm .depdirs
>  $ time make -j
>  real 0m23.734s user  0m11.228s sys   0m4.844s
>  
>  About 16 seconds, which is not a lot of savings. Now the difference from 
>  no -j to -j is a lot, but the difference between -j and -j  
>  is not a huge saving. This leads me back to over engineering the problem 
>  when ?-j? would 

[dpdk-dev] [PATCH] validate_abi: build faster by augmenting make with job count

[Wiles, Keith]

> On Jul 21, 2016, at 8:54 AM, Neil Horman  wrote:
> 
> On Wed, Jul 20, 2016 at 10:32:28PM +, Wiles, Keith wrote:
>> 
>>> On Jul 20, 2016, at 3:16 PM, Neil Horman  wrote:
>>> 
>>> On Wed, Jul 20, 2016 at 07:47:32PM +, Wiles, Keith wrote:
 
> On Jul 20, 2016, at 12:48 PM, Neil Horman  wrote:
> 
> On Wed, Jul 20, 2016 at 07:40:49PM +0200, Thomas Monjalon wrote:
>> 2016-07-20 13:09, Neil Horman:
>>> From: Neil Horman 
>>> 
>>> John Mcnamara and I were discussing enhacing the validate_abi script to 
>>> build
>>> the dpdk tree faster with multiple jobs.  Theres no reason not to do 
>>> it, so this
>>> implements that requirement.  It uses a MAKE_JOBS variable that can be 
>>> set by
>>> the user to limit the job count.  By default the job count is set to 
>>> the number
>>> of online cpus.
>> 
>> Please could you use the variable name DPDK_MAKE_JOBS?
>> This name is already used in scripts/test-build.sh.
>> 
> Sure
> 
>>> +if [ -z "$MAKE_JOBS" ]
>>> +then
>>> +   # This counts the number of cpus on the system
>>> +   MAKE_JOBS=`lscpu -p=cpu | grep -v "#" | wc -l`
>>> +fi
>> 
>> Is lscpu common enough?
>> 
> I'm not sure how to answer that.  lscpu is part of the util-linux 
> package, which
> is part of any base install.  Theres a variant for BSD, but I'm not sure 
> how
> common it is there.
> Neil
> 
>> Another acceptable default would be just "-j" without any number.
>> It would make the number of jobs unlimited.
 
 I think the best is just use -j as it tries to use the correct number of 
 jobs based on the number of cores, right?
 
>>> -j with no argument (or -j 0), is sort of, maybe what you want.  With 
>>> either of
>>> those options, make will just issue jobs as fast as it processes 
>>> dependencies.
>>> Dependent on how parallel the build is, that can lead to tons of waiting 
>>> process
>>> (i.e. more than your number of online cpus), which can actually hurt your 
>>> build
>>> time.
>> 
>> I read the manual and looked at the code, which supports your statement. (I 
>> think I had some statement on stack overflow and the last time I believe 
>> anything on the internet :-) I have not seen a lot of differences in compile 
>> times with -j on my system. Mostly I suspect it is the number of paths in 
>> the dependency, cores and memory on the system.
>> 
>> I have 72 lcores or 2 sockets, 18 cores per socket. Xeon 2.3Ghz cores.
>> 
>> $ export RTE_TARGET=x86_64-native-linuxapp-gcc 
>> 
>> $ time make install T=${RTE_TARGET}
>> real 0m59.445s user  0m27.344s sys   0m7.040s
>> 
>> $ time make install T=${RTE_TARGET} -j
>> real 0m26.584s user  0m14.380s sys   0m5.120s
>> 
>> # Remove the x86_64-native-linuxapp-gcc
>> 
>> $ time make install T=${RTE_TARGET} -j 72
>> real 0m23.454s user  0m10.832s sys   0m4.664s
>> 
>> $ time make install T=${RTE_TARGET} -j 8
>> real 0m23.812s user  0m10.672s sys   0m4.276s
>> 
>> cd x86_64-native-linuxapp-gcc
>> $ make clean
>> $ time make
>> real 0m28.539s user  0m9.820s sys0m3.620s
>> 
>> # Do a make clean between each build.
>> 
>> $ time make -j
>> real 0m7.217s user   0m6.532s sys0m2.332s
>> 
>> $ time make -j 8
>> real 0m8.256s user   0m6.472s sys0m2.456s
>> 
>> $ time make -j 72
>> real 0m6.866s user   0m6.184s sys0m2.216s
>> 
>> Just the real time numbers in the following table.
>> 
>> processes real Time   depdirs
>> no -j 59.4sYes
>>   -j 8 23.8sYes
>>  -j 7223.5sYes
>>-j   26.5sYes
>> 
>> no -j 28.5s No
>>   -j 8   8.2s No
>>  -j 72  6.8s No
>>-j 7.2s No
>> 
>> Looks like the depdirs build time on my system:
>> $ make clean -j
>> $ rm .depdirs
>> $ time make -j
>> real 0m23.734s user  0m11.228s sys   0m4.844s
>> 
>> About 16 seconds, which is not a lot of savings. Now the difference from no 
>> -j to -j is a lot, but the difference between -j and -j  is not a 
>> huge saving. This leads me back to over engineering the problem when ?-j? 
>> would work just as well here.
>> 
>> Even on my MacBook Pro i7 system the difference is not that much 1m8s 
>> without depdirs build for -j in a VirtualBox with all 4 cores 8G RAM. 
>> Compared to 1m13s with -j 4 option.
>> 
>> I just wonder if it makes a lot of sense to use cpuinfo in this given case 
>> if it turns out to be -j works with the 80% rule?
>> 
> It may, but that seems to be reason to me to just set DPDK_MAKE_JOBS=0, and
> you'll get that behavior

Just to be sure, ?make -j 0? is not a valid argument to the -j option. It looks 
like you have to do ?-j? or ?-j N? or no option where N != 0

I think we just use -j which gets us the 80% rule and the best performance 
without counting cores.

> 
> Neil
> 
>> On 

[dpdk-dev] [PATCH] validate_abi: build faster by augmenting make with job count

[Neil Horman]

On Thu, Jul 21, 2016 at 02:09:19PM +, Wiles, Keith wrote:
> 
> > On Jul 21, 2016, at 8:54 AM, Neil Horman  wrote:
> > 
> > On Wed, Jul 20, 2016 at 10:32:28PM +, Wiles, Keith wrote:
> >> 
> >>> On Jul 20, 2016, at 3:16 PM, Neil Horman  wrote:
> >>> 
> >>> On Wed, Jul 20, 2016 at 07:47:32PM +, Wiles, Keith wrote:
>  
> > On Jul 20, 2016, at 12:48 PM, Neil Horman  wrote:
> > 
> > On Wed, Jul 20, 2016 at 07:40:49PM +0200, Thomas Monjalon wrote:
> >> 2016-07-20 13:09, Neil Horman:
> >>> From: Neil Horman 
> >>> 
> >>> John Mcnamara and I were discussing enhacing the validate_abi script 
> >>> to build
> >>> the dpdk tree faster with multiple jobs.  Theres no reason not to do 
> >>> it, so this
> >>> implements that requirement.  It uses a MAKE_JOBS variable that can 
> >>> be set by
> >>> the user to limit the job count.  By default the job count is set to 
> >>> the number
> >>> of online cpus.
> >> 
> >> Please could you use the variable name DPDK_MAKE_JOBS?
> >> This name is already used in scripts/test-build.sh.
> >> 
> > Sure
> > 
> >>> +if [ -z "$MAKE_JOBS" ]
> >>> +then
> >>> + # This counts the number of cpus on the system
> >>> + MAKE_JOBS=`lscpu -p=cpu | grep -v "#" | wc -l`
> >>> +fi
> >> 
> >> Is lscpu common enough?
> >> 
> > I'm not sure how to answer that.  lscpu is part of the util-linux 
> > package, which
> > is part of any base install.  Theres a variant for BSD, but I'm not 
> > sure how
> > common it is there.
> > Neil
> > 
> >> Another acceptable default would be just "-j" without any number.
> >> It would make the number of jobs unlimited.
>  
>  I think the best is just use -j as it tries to use the correct number of 
>  jobs based on the number of cores, right?
>  
> >>> -j with no argument (or -j 0), is sort of, maybe what you want.  With 
> >>> either of
> >>> those options, make will just issue jobs as fast as it processes 
> >>> dependencies.
> >>> Dependent on how parallel the build is, that can lead to tons of waiting 
> >>> process
> >>> (i.e. more than your number of online cpus), which can actually hurt your 
> >>> build
> >>> time.
> >> 
> >> I read the manual and looked at the code, which supports your statement. 
> >> (I think I had some statement on stack overflow and the last time I 
> >> believe anything on the internet :-) I have not seen a lot of differences 
> >> in compile times with -j on my system. Mostly I suspect it is the number 
> >> of paths in the dependency, cores and memory on the system.
> >> 
> >> I have 72 lcores or 2 sockets, 18 cores per socket. Xeon 2.3Ghz cores.
> >> 
> >> $ export RTE_TARGET=x86_64-native-linuxapp-gcc 
> >> 
> >> $ time make install T=${RTE_TARGET}
> >> real   0m59.445s user  0m27.344s sys   0m7.040s
> >> 
> >> $ time make install T=${RTE_TARGET} -j
> >> real   0m26.584s user  0m14.380s sys   0m5.120s
> >> 
> >> # Remove the x86_64-native-linuxapp-gcc
> >> 
> >> $ time make install T=${RTE_TARGET} -j 72
> >> real   0m23.454s user  0m10.832s sys   0m4.664s
> >> 
> >> $ time make install T=${RTE_TARGET} -j 8
> >> real   0m23.812s user  0m10.672s sys   0m4.276s
> >> 
> >> cd x86_64-native-linuxapp-gcc
> >> $ make clean
> >> $ time make
> >> real   0m28.539s user  0m9.820s sys0m3.620s
> >> 
> >> # Do a make clean between each build.
> >> 
> >> $ time make -j
> >> real   0m7.217s user   0m6.532s sys0m2.332s
> >> 
> >> $ time make -j 8
> >> real   0m8.256s user   0m6.472s sys0m2.456s
> >> 
> >> $ time make -j 72
> >> real   0m6.866s user   0m6.184s sys0m2.216s
> >> 
> >> Just the real time numbers in the following table.
> >> 
> >> processes real Time   depdirs
> >> no -j 59.4sYes
> >>   -j 8 23.8sYes
> >>  -j 7223.5sYes
> >>-j   26.5sYes
> >> 
> >> no -j 28.5s No
> >>   -j 8   8.2s No
> >>  -j 72  6.8s No
> >>-j 7.2s No
> >> 
> >> Looks like the depdirs build time on my system:
> >> $ make clean -j
> >> $ rm .depdirs
> >> $ time make -j
> >> real   0m23.734s user  0m11.228s sys   0m4.844s
> >> 
> >> About 16 seconds, which is not a lot of savings. Now the difference from 
> >> no -j to -j is a lot, but the difference between -j and -j  is 
> >> not a huge saving. This leads me back to over engineering the problem when 
> >> ?-j? would work just as well here.
> >> 
> >> Even on my MacBook Pro i7 system the difference is not that much 1m8s 
> >> without depdirs build for -j in a VirtualBox with all 4 cores 8G RAM. 
> >> Compared to 1m13s with -j 4 option.
> >> 
> >> I just wonder if it makes a lot of sense to use cpuinfo in this given case 
> >> if it turns out to be -j works with the 80% rule?
> 

[dpdk-dev] [PATCH] validate_abi: build faster by augmenting make with job count

[Neil Horman]

On Wed, Jul 20, 2016 at 10:32:28PM +, Wiles, Keith wrote:
> 
> > On Jul 20, 2016, at 3:16 PM, Neil Horman  wrote:
> > 
> > On Wed, Jul 20, 2016 at 07:47:32PM +, Wiles, Keith wrote:
> >> 
> >>> On Jul 20, 2016, at 12:48 PM, Neil Horman  wrote:
> >>> 
> >>> On Wed, Jul 20, 2016 at 07:40:49PM +0200, Thomas Monjalon wrote:
>  2016-07-20 13:09, Neil Horman:
> > From: Neil Horman 
> > 
> > John Mcnamara and I were discussing enhacing the validate_abi script to 
> > build
> > the dpdk tree faster with multiple jobs.  Theres no reason not to do 
> > it, so this
> > implements that requirement.  It uses a MAKE_JOBS variable that can be 
> > set by
> > the user to limit the job count.  By default the job count is set to 
> > the number
> > of online cpus.
>  
>  Please could you use the variable name DPDK_MAKE_JOBS?
>  This name is already used in scripts/test-build.sh.
>  
> >>> Sure
> >>> 
> > +if [ -z "$MAKE_JOBS" ]
> > +then
> > +   # This counts the number of cpus on the system
> > +   MAKE_JOBS=`lscpu -p=cpu | grep -v "#" | wc -l`
> > +fi
>  
>  Is lscpu common enough?
>  
> >>> I'm not sure how to answer that.  lscpu is part of the util-linux 
> >>> package, which
> >>> is part of any base install.  Theres a variant for BSD, but I'm not sure 
> >>> how
> >>> common it is there.
> >>> Neil
> >>> 
>  Another acceptable default would be just "-j" without any number.
>  It would make the number of jobs unlimited.
> >> 
> >> I think the best is just use -j as it tries to use the correct number of 
> >> jobs based on the number of cores, right?
> >> 
> > -j with no argument (or -j 0), is sort of, maybe what you want.  With 
> > either of
> > those options, make will just issue jobs as fast as it processes 
> > dependencies.
> > Dependent on how parallel the build is, that can lead to tons of waiting 
> > process
> > (i.e. more than your number of online cpus), which can actually hurt your 
> > build
> > time.
> 
> I read the manual and looked at the code, which supports your statement. (I 
> think I had some statement on stack overflow and the last time I believe 
> anything on the internet :-) I have not seen a lot of differences in compile 
> times with -j on my system. Mostly I suspect it is the number of paths in the 
> dependency, cores and memory on the system.
> 
> I have 72 lcores or 2 sockets, 18 cores per socket. Xeon 2.3Ghz cores.
> 
> $ export RTE_TARGET=x86_64-native-linuxapp-gcc 
> 
> $ time make install T=${RTE_TARGET}
> real  0m59.445s user  0m27.344s sys   0m7.040s
> 
> $ time make install T=${RTE_TARGET} -j
> real  0m26.584s user  0m14.380s sys   0m5.120s
> 
> # Remove the x86_64-native-linuxapp-gcc
> 
> $ time make install T=${RTE_TARGET} -j 72
> real  0m23.454s user  0m10.832s sys   0m4.664s
> 
> $ time make install T=${RTE_TARGET} -j 8
> real  0m23.812s user  0m10.672s sys   0m4.276s
> 
> cd x86_64-native-linuxapp-gcc
> $ make clean
> $ time make
> real  0m28.539s user  0m9.820s sys0m3.620s
> 
> # Do a make clean between each build.
> 
> $ time make -j
> real  0m7.217s user   0m6.532s sys0m2.332s
> 
> $ time make -j 8
> real  0m8.256s user   0m6.472s sys0m2.456s
> 
> $ time make -j 72
> real  0m6.866s user   0m6.184s sys0m2.216s
> 
> Just the real time numbers in the following table.
> 
> processes real Time   depdirs
>  no -j 59.4sYes
>-j 8 23.8sYes
>   -j 7223.5sYes
> -j   26.5sYes
> 
>  no -j 28.5s No
>-j 8   8.2s No
>   -j 72  6.8s No
> -j 7.2s No
> 
> Looks like the depdirs build time on my system:
> $ make clean -j
> $ rm .depdirs
> $ time make -j
> real  0m23.734s user  0m11.228s sys   0m4.844s
> 
> About 16 seconds, which is not a lot of savings. Now the difference from no 
> -j to -j is a lot, but the difference between -j and -j  is not a 
> huge saving. This leads me back to over engineering the problem when ?-j? 
> would work just as well here.
> 
> Even on my MacBook Pro i7 system the difference is not that much 1m8s without 
> depdirs build for -j in a VirtualBox with all 4 cores 8G RAM. Compared to 
> 1m13s with -j 4 option.
> 
> I just wonder if it makes a lot of sense to use cpuinfo in this given case if 
> it turns out to be -j works with the 80% rule?
> 
It may, but that seems to be reason to me to just set DPDK_MAKE_JOBS=0, and
you'll get that behavior

Neil

> On some other project with a lot more files like the FreeBSD or Linux distro, 
> yes it would make a fair amount of real time difference.
> 
> Keith
> 
> > 
> > While its fine in los of cases, its not always fine, and with this
> > implementation you can still opt in to that behavior by setting 
> > DPDK_MAKE_JOBS=0
> > 
> > Neil
> > 
> >> 
> 

[dpdk-dev] [PATCH] validate_abi: build faster by augmenting make with job count

[Wiles, Keith]

> On Jul 20, 2016, at 3:16 PM, Neil Horman  wrote:
> 
> On Wed, Jul 20, 2016 at 07:47:32PM +, Wiles, Keith wrote:
>> 
>>> On Jul 20, 2016, at 12:48 PM, Neil Horman  wrote:
>>> 
>>> On Wed, Jul 20, 2016 at 07:40:49PM +0200, Thomas Monjalon wrote:
 2016-07-20 13:09, Neil Horman:
> From: Neil Horman 
> 
> John Mcnamara and I were discussing enhacing the validate_abi script to 
> build
> the dpdk tree faster with multiple jobs.  Theres no reason not to do it, 
> so this
> implements that requirement.  It uses a MAKE_JOBS variable that can be 
> set by
> the user to limit the job count.  By default the job count is set to the 
> number
> of online cpus.
 
 Please could you use the variable name DPDK_MAKE_JOBS?
 This name is already used in scripts/test-build.sh.
 
>>> Sure
>>> 
> +if [ -z "$MAKE_JOBS" ]
> +then
> + # This counts the number of cpus on the system
> + MAKE_JOBS=`lscpu -p=cpu | grep -v "#" | wc -l`
> +fi
 
 Is lscpu common enough?
 
>>> I'm not sure how to answer that.  lscpu is part of the util-linux package, 
>>> which
>>> is part of any base install.  Theres a variant for BSD, but I'm not sure how
>>> common it is there.
>>> Neil
>>> 
 Another acceptable default would be just "-j" without any number.
 It would make the number of jobs unlimited.
>> 
>> I think the best is just use -j as it tries to use the correct number of 
>> jobs based on the number of cores, right?
>> 
> -j with no argument (or -j 0), is sort of, maybe what you want.  With either 
> of
> those options, make will just issue jobs as fast as it processes dependencies.
> Dependent on how parallel the build is, that can lead to tons of waiting 
> process
> (i.e. more than your number of online cpus), which can actually hurt your 
> build
> time.

I read the manual and looked at the code, which supports your statement. (I 
think I had some statement on stack overflow and the last time I believe 
anything on the internet :-) I have not seen a lot of differences in compile 
times with -j on my system. Mostly I suspect it is the number of paths in the 
dependency, cores and memory on the system.

I have 72 lcores or 2 sockets, 18 cores per socket. Xeon 2.3Ghz cores.

$ export RTE_TARGET=x86_64-native-linuxapp-gcc 

$ time make install T=${RTE_TARGET}
real0m59.445s user  0m27.344s sys   0m7.040s

$ time make install T=${RTE_TARGET} -j
real0m26.584s user  0m14.380s sys   0m5.120s

# Remove the x86_64-native-linuxapp-gcc

$ time make install T=${RTE_TARGET} -j 72
real0m23.454s user  0m10.832s sys   0m4.664s

$ time make install T=${RTE_TARGET} -j 8
real0m23.812s user  0m10.672s sys   0m4.276s

cd x86_64-native-linuxapp-gcc
$ make clean
$ time make
real0m28.539s user  0m9.820s sys0m3.620s

# Do a make clean between each build.

$ time make -j
real0m7.217s user   0m6.532s sys0m2.332s

$ time make -j 8
real0m8.256s user   0m6.472s sys0m2.456s

$ time make -j 72
real0m6.866s user   0m6.184s sys0m2.216s

Just the real time numbers in the following table.

processes real Time   depdirs
 no -j 59.4sYes
   -j 8 23.8sYes
  -j 7223.5sYes
-j   26.5sYes

 no -j 28.5s No
   -j 8   8.2s No
  -j 72  6.8s No
-j 7.2s No

Looks like the depdirs build time on my system:
$ make clean -j
$ rm .depdirs
$ time make -j
real0m23.734s user  0m11.228s sys   0m4.844s

About 16 seconds, which is not a lot of savings. Now the difference from no -j 
to -j is a lot, but the difference between -j and -j  is not a huge 
saving. This leads me back to over engineering the problem when ?-j? would work 
just as well here.

Even on my MacBook Pro i7 system the difference is not that much 1m8s without 
depdirs build for -j in a VirtualBox with all 4 cores 8G RAM. Compared to 1m13s 
with -j 4 option.

I just wonder if it makes a lot of sense to use cpuinfo in this given case if 
it turns out to be -j works with the 80% rule?

On some other project with a lot more files like the FreeBSD or Linux distro, 
yes it would make a fair amount of real time difference.

Keith

> 
> While its fine in los of cases, its not always fine, and with this
> implementation you can still opt in to that behavior by setting 
> DPDK_MAKE_JOBS=0
> 
> Neil
> 
>> 

[dpdk-dev] [PATCH] validate_abi: build faster by augmenting make with job count

[Thomas Monjalon]

2016-07-20 19:47, Wiles, Keith:
> On Jul 20, 2016, at 12:48 PM, Neil Horman  wrote:
> > On Wed, Jul 20, 2016 at 07:40:49PM +0200, Thomas Monjalon wrote:
> >> 2016-07-20 13:09, Neil Horman:
> >>> From: Neil Horman 
> >>> +if [ -z "$MAKE_JOBS" ]
> >>> +then
> >>> + # This counts the number of cpus on the system
> >>> + MAKE_JOBS=`lscpu -p=cpu | grep -v "#" | wc -l`
> >>> +fi
> >> 
> >> Is lscpu common enough?
> >> 
> > I'm not sure how to answer that.  lscpu is part of the util-linux package, 
> > which
> > is part of any base install.  Theres a variant for BSD, but I'm not sure how
> > common it is there.
> > Neil
> > 
> >> Another acceptable default would be just "-j" without any number.
> >> It would make the number of jobs unlimited.
> 
> I think the best is just use -j as it tries to use the correct number of jobs 
> based on the number of cores, right?

No Keith, -j alone use as much jobs as it can create, i.e. much more than
the number of CPUs.
I have no measure but I remember it is less efficient than giving a number
based on available CPUs (with a multiply factor to avoid idling between jobs).
For a default value, both approaches are fine.

[dpdk-dev] [PATCH] validate_abi: build faster by augmenting make with job count

[Wiles, Keith]

> On Jul 20, 2016, at 12:48 PM, Neil Horman  wrote:
> 
> On Wed, Jul 20, 2016 at 07:40:49PM +0200, Thomas Monjalon wrote:
>> 2016-07-20 13:09, Neil Horman:
>>> From: Neil Horman 
>>> 
>>> John Mcnamara and I were discussing enhacing the validate_abi script to 
>>> build
>>> the dpdk tree faster with multiple jobs.  Theres no reason not to do it, so 
>>> this
>>> implements that requirement.  It uses a MAKE_JOBS variable that can be set 
>>> by
>>> the user to limit the job count.  By default the job count is set to the 
>>> number
>>> of online cpus.
>> 
>> Please could you use the variable name DPDK_MAKE_JOBS?
>> This name is already used in scripts/test-build.sh.
>> 
> Sure
> 
>>> +if [ -z "$MAKE_JOBS" ]
>>> +then
>>> +   # This counts the number of cpus on the system
>>> +   MAKE_JOBS=`lscpu -p=cpu | grep -v "#" | wc -l`
>>> +fi
>> 
>> Is lscpu common enough?
>> 
> I'm not sure how to answer that.  lscpu is part of the util-linux package, 
> which
> is part of any base install.  Theres a variant for BSD, but I'm not sure how
> common it is there.
> Neil
> 
>> Another acceptable default would be just "-j" without any number.
>> It would make the number of jobs unlimited.

I think the best is just use -j as it tries to use the correct number of jobs 
based on the number of cores, right?

[dpdk-dev] [PATCH] validate_abi: build faster by augmenting make with job count

[Thomas Monjalon]

2016-07-20 13:09, Neil Horman:
> From: Neil Horman 
> 
> John Mcnamara and I were discussing enhacing the validate_abi script to build
> the dpdk tree faster with multiple jobs.  Theres no reason not to do it, so 
> this
> implements that requirement.  It uses a MAKE_JOBS variable that can be set by
> the user to limit the job count.  By default the job count is set to the 
> number
> of online cpus.

Please could you use the variable name DPDK_MAKE_JOBS?
This name is already used in scripts/test-build.sh.

> +if [ -z "$MAKE_JOBS" ]
> +then
> + # This counts the number of cpus on the system
> + MAKE_JOBS=`lscpu -p=cpu | grep -v "#" | wc -l`
> +fi

Is lscpu common enough?

Another acceptable default would be just "-j" without any number.
It would make the number of jobs unlimited.

[dpdk-dev] [PATCH] validate_abi: build faster by augmenting make with job count

[Neil Horman]

On Wed, Jul 20, 2016 at 07:47:32PM +, Wiles, Keith wrote:
> 
> > On Jul 20, 2016, at 12:48 PM, Neil Horman  wrote:
> > 
> > On Wed, Jul 20, 2016 at 07:40:49PM +0200, Thomas Monjalon wrote:
> >> 2016-07-20 13:09, Neil Horman:
> >>> From: Neil Horman 
> >>> 
> >>> John Mcnamara and I were discussing enhacing the validate_abi script to 
> >>> build
> >>> the dpdk tree faster with multiple jobs.  Theres no reason not to do it, 
> >>> so this
> >>> implements that requirement.  It uses a MAKE_JOBS variable that can be 
> >>> set by
> >>> the user to limit the job count.  By default the job count is set to the 
> >>> number
> >>> of online cpus.
> >> 
> >> Please could you use the variable name DPDK_MAKE_JOBS?
> >> This name is already used in scripts/test-build.sh.
> >> 
> > Sure
> > 
> >>> +if [ -z "$MAKE_JOBS" ]
> >>> +then
> >>> + # This counts the number of cpus on the system
> >>> + MAKE_JOBS=`lscpu -p=cpu | grep -v "#" | wc -l`
> >>> +fi
> >> 
> >> Is lscpu common enough?
> >> 
> > I'm not sure how to answer that.  lscpu is part of the util-linux package, 
> > which
> > is part of any base install.  Theres a variant for BSD, but I'm not sure how
> > common it is there.
> > Neil
> > 
> >> Another acceptable default would be just "-j" without any number.
> >> It would make the number of jobs unlimited.
> 
> I think the best is just use -j as it tries to use the correct number of jobs 
> based on the number of cores, right?
> 
-j with no argument (or -j 0), is sort of, maybe what you want.  With either of
those options, make will just issue jobs as fast as it processes dependencies.
Dependent on how parallel the build is, that can lead to tons of waiting process
(i.e. more than your number of online cpus), which can actually hurt your build
time.

While its fine in los of cases, its not always fine, and with this
implementation you can still opt in to that behavior by setting DPDK_MAKE_JOBS=0

Neil

> 

[dpdk-dev] [PATCH] validate_abi: build faster by augmenting make with job count

[Neil Horman]

On Wed, Jul 20, 2016 at 07:40:49PM +0200, Thomas Monjalon wrote:
> 2016-07-20 13:09, Neil Horman:
> > From: Neil Horman 
> > 
> > John Mcnamara and I were discussing enhacing the validate_abi script to 
> > build
> > the dpdk tree faster with multiple jobs.  Theres no reason not to do it, so 
> > this
> > implements that requirement.  It uses a MAKE_JOBS variable that can be set 
> > by
> > the user to limit the job count.  By default the job count is set to the 
> > number
> > of online cpus.
> 
> Please could you use the variable name DPDK_MAKE_JOBS?
> This name is already used in scripts/test-build.sh.
> 
Sure

> > +if [ -z "$MAKE_JOBS" ]
> > +then
> > +   # This counts the number of cpus on the system
> > +   MAKE_JOBS=`lscpu -p=cpu | grep -v "#" | wc -l`
> > +fi
> 
> Is lscpu common enough?
> 
I'm not sure how to answer that.  lscpu is part of the util-linux package, which
is part of any base install.  Theres a variant for BSD, but I'm not sure how
common it is there.
Neil

> Another acceptable default would be just "-j" without any number.
> It would make the number of jobs unlimited.

[dpdk-dev] [PATCH] validate_abi: build faster by augmenting make with job count

[Neil Horman]

From: Neil Horman 

John Mcnamara and I were discussing enhacing the validate_abi script to build
the dpdk tree faster with multiple jobs.  Theres no reason not to do it, so this
implements that requirement.  It uses a MAKE_JOBS variable that can be set by
the user to limit the job count.  By default the job count is set to the number
of online cpus.

Signed-off-by: Neil Horman 
CC: Thomas Monjalon 
CC: "Mcnamara, John" 
---
 scripts/validate-abi.sh | 10 --
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/scripts/validate-abi.sh b/scripts/validate-abi.sh
index c36ad61..1c9627b 100755
--- a/scripts/validate-abi.sh
+++ b/scripts/validate-abi.sh
@@ -97,6 +97,12 @@ fixup_config() {
 #trap on ctrl-c to clean up
 trap cleanup_and_exit SIGINT

+if [ -z "$MAKE_JOBS" ]
+then
+   # This counts the number of cpus on the system
+   MAKE_JOBS=`lscpu -p=cpu | grep -v "#" | wc -l`
+fi
+
 #Save the current branch
 CURRENT_BRANCH=`git branch | grep \* | cut -d' ' -f2`

@@ -183,7 +189,7 @@ log "INFO" "Configuring DPDK $TAG1"
 make config T=$TARGET O=$TARGET > $VERBOSE 2>&1

 log "INFO" "Building DPDK $TAG1. This might take a moment"
-make O=$TARGET > $VERBOSE 2>&1
+make -j$MAKE_JOBS O=$TARGET > $VERBOSE 2>&1

 if [ $? -ne 0 ]
 then
@@ -214,7 +220,7 @@ log "INFO" "Configuring DPDK $TAG2"
 make config T=$TARGET O=$TARGET > $VERBOSE 2>&1

 log "INFO" "Building DPDK $TAG2. This might take a moment"
-make O=$TARGET > $VERBOSE 2>&1
+make -j$MAKE_JOBS O=$TARGET > $VERBOSE 2>&1

 if [ $? -ne 0 ]
 then
-- 
2.5.5