Re: Jenkins with Parallel Builds?

[Simon Richter]

Hi,

On Wed, Jan 15, 2020 at 02:38:26PM +, David Aldrich wrote:

> We build large C++ projects using Jenkins for Linux platforms.  These build
> jobs currently run on a single executor.  I want 'make -j' or ninja to
> compile concurrently on several cores to speed up the build time.  How can
> I cause a job to utilise multiple executors on an agent in this way?

If you want to go for a 1:1 executor to CPU mapping, then using -j inside
the build and combining it with the Heavy Job plugin to allocate more CPUs
is indeed the way to go.

The downside is that it will wait until that many executors are available,
and IIRC there is an anti-starvation mechanism that also blocks other jobs
from executing on the chosen node.

Depending on your setup, you might as well have the best result with a
setup like "four executors, and all jobs use "-j$((($(nproc)+3)/4)" or
something similar.

For the most flexible result, I suggest defining an environment variable in
the agent config that defines the -j option to be used on that node, then
you have both the number of executors and the number of threads per
executor in the agent config, and if you see that it doesn't work
optimally, you can tune there.

Two or four CPUs per executor are good if you have many builds, while two
or four executors with half/quarter of the CPUs each are good if you build
a massive project seldom.

The Heavy Jobs plugin is for mixing strategies -- e.g. you'd define a -j4
job for incremental builds that run quickly either way except if there is a
big change, and a heavyweight job that takes over half the node for the
daily full rebuild/warning generation/... task.

   Simon

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/2020011514.GA23937%40psi5.com.

Re: Jenkins with Parallel Builds?

[Dirk Heinrichs]

Am Mittwoch, den 15.01.2020, 14:38 + schrieb David Aldrich:

We build large C++ projects using Jenkins for Linux platforms.  These build 
jobs currently run on a single executor.  I want 'make -j' or ninja to compile 
concurrently on several cores to speed up the build time.  How can I cause a 
job to utilise multiple executors on an agent in this way?

These are two completely different things. The number of executers defines how 
man Jenkins jobs an agent can run in parallel. If you want to run your build 
(as executed by ONE job) as a parallel build, then you just need to specify the 
level of concurrency inside that single job, just like you would do if it was 
executed manually from within a shell (don't know whether Jenkins has a "make" 
plugin).

This is also how we do it. All build nodes only have one Jenkins executor to 
ensure we only run one job at a time on any given node. But the builds executed 
by these jobs can then utilize all the CPU cores in that node.

HTH...

Dirk

--

Dirk Heinrichs
Senior Systems Engineer, Delivery Pipeline
OpenText ™ Discovery | Recommind
Phone: +49 2226 15966 18
Email: dhein...@opentext.com
Website: www.recommind.de
Recommind GmbH, Von-Liebig-Straße 1, 53359 Rheinbach
Vertretungsberechtigte Geschäftsführer Gordon Davies, Madhu Ranganathan, 
Christian Waida, Registergericht Amtsgericht Bonn, Registernummer HRB 10646
This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy this e-mail. Any unauthorized 
copying, disclosure or distribution of the material in this e-mail is strictly 
forbidden
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. 
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten 
haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. 
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail sind nicht 
gestattet.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/f2dfe24cf54d69660e1c45a508366b6d850853c3.camel%40opentext.com.