Hi,
On Thu, Dec 03, 2020 at 05:48:16PM +0000, Ross Burton wrote:
> Hi,
>
> Currently, BB_NUMBER_THREADS and PARALLEL_MAKE use the number of cores
> available unless told otherwise. This was a good idea six years
> ago[1] but some modern machines are moving to very large core counts.
>
> For example, 88 core dual Xeons are fairly common. A ThunderX2 has 256
> cores (2 sockets, 4 hyperthreads per physical core). The Ampere Altra
> is dual socket 2*80=160 cores.
>
> At this level of parallelisation the sheer amount of I/O from the
> unpack storm is quite excessive. As a strawman argument, I propose a
> hard cap to the default BB_NUMBER_THREADS of -- and I'm literally
> making up numbers here -- 32. Maybe 64. Comments?
Number of cores is far from sufficient in real world. Amount of physical RAM
should be
taken into account too. I'd say 2 Gb of physical RAM per thread should be
available to
compile and link modern C++ SW to avoid out-of-memory killer from kicking in.
Here is one algorithm which avoids oom killer in our case:
mem = get_mem_total()
cpus = get_number_cpus()
mem_cpus = (mem * 1.0) / cpus
if cpus == 1:
# In case of a single CPU, don't parallelize
self.bb_number_threads = 1
self.parallel_make = make_j(1)
elif mem_cpus > 8:
self.bb_number_threads = cpus
self.parallel_make = make_j(cpus)
elif mem_cpus >= 4:
self.bb_number_threads = cpus
self.parallel_make = make_j(divide_cpus(cpus, 2))
elif mem_cpus >= 2:
self.bb_number_threads = divide_cpus(cpus, 2)
self.parallel_make = make_j(divide_cpus(cpus, 2))
else:
self.bb_number_threads = divide_cpus(cpus, 2)
self.parallel_make = make_j(divide_cpus(cpus, 4))
Cheers,
-Mikko
>
> Cheers,
> Ross
>
> [1]
> http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=1529ef0504542145f2b81b2dba4bcc81d5dac96e
>
>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#145286):
https://lists.openembedded.org/g/openembedded-core/message/145286
Mute This Topic: https://lists.openembedded.org/mt/78690216/21656
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
