Re: [OE-core] [PATCH 1/1] bitbake.conf: Set ZSTD_THREADS to half of cpu number

[Robert Yang]

Hi RP,

On 11/9/21 6:26 PM, Richard Purdie wrote:

On Tue, 2021-11-09 at 10:56 +0100, Alexander Kanavin wrote:

On Tue, 9 Nov 2021 at 10:51, Robert Yang  wrote:

Maybe, but once we start doing tweaks like this, where do they stop? Hey,

I'd

I think that it's not only a performance tweak, but this should be
considered
as
a bug since it would be failed to build on powerful machine, which isn't
good
for oe-core's user experience.




I managed to bring down a powerful machine just yesterday by building webkit
and llvm together at the same time. They together ate all the RAM, and someone
in the office had to go and reset the box.
Do I see it as a bug? Absolutely not: it's in fact my fault for not ensuring
system resource consumption doesn't overwhelm the machine. This is the same:
if you do not have RAM to match the cores on your specific rig, add RAM or
limit the threads.


I can see both sides of this. If we had a pool for all the threads and the
pieces of the system all used the same pool, it would work well as implemented.
Sadly the bitbake threads, the make processes and now the compression threads
are all separate.

We do see the autobuilder break due to load issues too and on that we have
scaled back some of the defaults but not ZSTD as yet that I recall.

As such I think limiting the upper value on number of threads may be a good
compromise, both for compression and maybe for parallel make too.


What should we do next, please? Limit bb thread + parallel make + zstd thread?
Or just limit zstd's thread since other people may concern about the
performance.

// Robert



Cheers,

Richard


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#158147): 
https://lists.openembedded.org/g/openembedded-core/message/158147
Mute This Topic: https://lists.openembedded.org/mt/86926962/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH 1/1] bitbake.conf: Set ZSTD_THREADS to half of cpu number

[Richard Purdie]

On Tue, 2021-11-09 at 10:56 +0100, Alexander Kanavin wrote:
> On Tue, 9 Nov 2021 at 10:51, Robert Yang  wrote:
> > > Maybe, but once we start doing tweaks like this, where do they stop? Hey,
> > I'd 
> > 
> > I think that it's not only a performance tweak, but this should be
> > considered
> > as
> > a bug since it would be failed to build on powerful machine, which isn't
> > good
> > for oe-core's user experience.
> > 
> 
> 
> I managed to bring down a powerful machine just yesterday by building webkit
> and llvm together at the same time. They together ate all the RAM, and someone
> in the office had to go and reset the box.
> Do I see it as a bug? Absolutely not: it's in fact my fault for not ensuring
> system resource consumption doesn't overwhelm the machine. This is the same:
> if you do not have RAM to match the cores on your specific rig, add RAM or
> limit the threads.

I can see both sides of this. If we had a pool for all the threads and the
pieces of the system all used the same pool, it would work well as implemented.
Sadly the bitbake threads, the make processes and now the compression threads
are all separate.

We do see the autobuilder break due to load issues too and on that we have
scaled back some of the defaults but not ZSTD as yet that I recall.

As such I think limiting the upper value on number of threads may be a good
compromise, both for compression and maybe for parallel make too.

Cheers,

Richard


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#158009): 
https://lists.openembedded.org/g/openembedded-core/message/158009
Mute This Topic: https://lists.openembedded.org/mt/86926962/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH 1/1] bitbake.conf: Set ZSTD_THREADS to half of cpu number

[Alexander Kanavin]

On Tue, 9 Nov 2021 at 10:51, Robert Yang  wrote:

> > Maybe, but once we start doing tweaks like this, where do they stop?
> Hey, I'd
>
> I think that it's not only a performance tweak, but this should be
> considered as
> a bug since it would be failed to build on powerful machine, which isn't
> good
> for oe-core's user experience.
>

I managed to bring down a powerful machine just yesterday by building
webkit and llvm together at the same time. They together ate all the RAM,
and someone in the office had to go and reset the box.
Do I see it as a bug? Absolutely not: it's in fact my fault for not
ensuring system resource consumption doesn't overwhelm the machine. This is
the same: if you do not have RAM to match the cores on your specific rig,
add RAM or limit the threads.

Alex

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#158008): 
https://lists.openembedded.org/g/openembedded-core/message/158008
Mute This Topic: https://lists.openembedded.org/mt/86926962/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH 1/1] bitbake.conf: Set ZSTD_THREADS to half of cpu number

[Robert Yang]

Hi Alexander,

On 11/9/21 5:44 PM, Alexander Kanavin wrote:
Maybe, but once we start doing tweaks like this, where do they stop? Hey, I'd 


I think that it's not only a performance tweak, but this should be considered as
a bug since it would be failed to build on powerful machine, which isn't good
for oe-core's user experience.

// Robert

like to limit llvm and webkit threads too, they eat disproportionally large 
amounts of RAM compared to everything else! I'd rather err on the side of 
simple, striaghtforward, universal defaults, and having better documentation for 
tweaking system resource consumption. Each build system is different.


Alex

On Tue, 9 Nov 2021 at 10:40, Robert Yang > wrote:




On 11/9/21 4:41 PM, Konrad Weihmann wrote:
 >
 >
 > On 09.11.21 09:48, Robert Yang wrote:
 >> The original value is very easy to cause do_packge error when cpu 
number is
 >> larger, for example, 128 cores and 512G mem:
 >>
 >> error: create archive failed: cpio: write failed - Cannot allocate 
memory"
 >>
 >> Set the ZSTD_THREADS to half of the CPU number can avoid the error in my
 >> testing.
 >>
 >> Signed-off-by: Robert Yang mailto:liezhi.y...@windriver.com>>
 >> ---
 >>   meta/conf/bitbake.conf | 2 +-
 >>   1 file changed, 1 insertion(+), 1 deletion(-)
 >>
 >> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
 >> index 71c1e52ad6..46ebf5113f 100644
 >> --- a/meta/conf/bitbake.conf
 >> +++ b/meta/conf/bitbake.conf
 >> @@ -833,7 +833,7 @@ XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT}
 >> --threads=${XZ_THREADS}"
 >>   XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"
 >>   # Default parallelism for zstd
 >> -ZSTD_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
 >> +ZSTD_THREADS ?= "${@int(oe.utils.cpu_count(at_least=4)/2)}"
 >
 > Then why not just limit it for the large setups you are referring to in 
the
 > example, for instance like
 >
 > ZSTD_THREADS ?= "${@min(int(oe.utils.cpu_count(at_least=4)),  value of your choice>)}"

This sounds like a good choice.

 >
 > BTW this can be also done in your local.conf - as Alex already said,
there is

Set it in my local.conf can make it work, but it isn't good for oe-core's 
OOBE
since the issue still exists.

 > simply no reason to make it slower for everyone, while also half the
threads of

The whole build contains a lot of tasks, for lower cpu cores, the total
resources are fixed, so when zstd uses less mem, other tasks can use more.

Limit zstd's thread doesn't make the build slower, but faster in my testing,
it would be great if you can help test it. The simple testing commands are:

# Make everything ready:
$ bitbake linux-yocto

# Before the patch, set ZSTD_THREADS to 128 and run 6 times:
$ for i in `seq 0 5`; do time bitbake linux-yocto -cpackage_write_rpm -f >
before_$i.log; done

Note, the time will be printed on the screen, not the before.log, the 
before.log
is for strip the logs and then we can read the time easier.

What I get is:
real    2m12.079s
real    2m0.177s
real    1m52.426s
real    2m3.396s
real    2m16.018s
real    1m58.595s

Drop the first build time since it *may* contain parsing time, and for the 
last
five builds:

Total: 609 seconds
Average: 609 / 5.0 = 121.8


# After the patch, set ZSTD_THREADS to 64, and run 6 times:
$ for i in `seq 0 5`; do time bitbake linux-yocto -cpackage_write_rpm -f >
after_$i.log; done

What I get is:
real    1m50.017s
real    1m50.400s
real    1m53.174s
real    2m4.817s
real    1m53.476s
real    1m56.794s

Drop the first build time since it *may* contain parsing time, and for the 
last
five builds:

Total: 576 seconds
Average: 576 / 5.0 = 115.2

So the smaller number is faster than the larger one.

// Robert

 > a 128 core machine could be a trouble some setup.
 >
 > Last time I had this issue (with XZ back then) I used
 > "${@min(int(oe.utils.cpu_count(at_least=4)), 20}"
 >
 >>   ZSTD_THREADS[vardepvalue] = "1"
 >>   # Limit the number of threads that OpenMP libraries will use.
Otherwise they
 >>
 >>
 >>
 >>
 >>




-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#158007): 
https://lists.openembedded.org/g/openembedded-core/message/158007
Mute This Topic: https://lists.openembedded.org/mt/86926962/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH 1/1] bitbake.conf: Set ZSTD_THREADS to half of cpu number

[Robert Yang]

Hi Alexander,

On 11/9/21 4:21 PM, Alexander Kanavin wrote:
But does this mean we need to globally restrict this and make things twice 
slower for everyone? I'm not sure.


Please see my reply to Konrad, it doesn't cause any slower, but faster in my 
testing.


BTW, the XZ_MEMLIMIT is 50% for command xz, but there isn't a similar option for
zstd, so I have to limit ZSTD_THREADS.

// Robert




Alex

On Tue, 9 Nov 2021 at 09:12, Robert Yang > wrote:


The original value is very easy to cause do_packge error when cpu number is
larger, for example, 128 cores and 512G mem:

error: create archive failed: cpio: write failed - Cannot allocate memory"

Set the ZSTD_THREADS to half of the CPU number can avoid the error in my
testing.

Signed-off-by: Robert Yang mailto:liezhi.y...@windriver.com>>
---
  meta/conf/bitbake.conf | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 71c1e52ad6..46ebf5113f 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -833,7 +833,7 @@ XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT}
--threads=${XZ_THREADS}"
  XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"

  # Default parallelism for zstd
-ZSTD_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
+ZSTD_THREADS ?= "${@int(oe.utils.cpu_count(at_least=4)/2)}"
  ZSTD_THREADS[vardepvalue] = "1"

  # Limit the number of threads that OpenMP libraries will use. Otherwise 
they
-- 
2.17.1






-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#158006): 
https://lists.openembedded.org/g/openembedded-core/message/158006
Mute This Topic: https://lists.openembedded.org/mt/86926962/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH 1/1] bitbake.conf: Set ZSTD_THREADS to half of cpu number

[Alexander Kanavin]

Maybe, but once we start doing tweaks like this, where do they stop? Hey,
I'd like to limit llvm and webkit threads too, they eat disproportionally
large amounts of RAM compared to everything else! I'd rather err on the
side of simple, striaghtforward, universal defaults, and having better
documentation for tweaking system resource consumption. Each build system
is different.

Alex

On Tue, 9 Nov 2021 at 10:40, Robert Yang  wrote:

>
>
> On 11/9/21 4:41 PM, Konrad Weihmann wrote:
> >
> >
> > On 09.11.21 09:48, Robert Yang wrote:
> >> The original value is very easy to cause do_packge error when cpu
> number is
> >> larger, for example, 128 cores and 512G mem:
> >>
> >> error: create archive failed: cpio: write failed - Cannot allocate
> memory"
> >>
> >> Set the ZSTD_THREADS to half of the CPU number can avoid the error in my
> >> testing.
> >>
> >> Signed-off-by: Robert Yang 
> >> ---
> >>   meta/conf/bitbake.conf | 2 +-
> >>   1 file changed, 1 insertion(+), 1 deletion(-)
> >>
> >> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> >> index 71c1e52ad6..46ebf5113f 100644
> >> --- a/meta/conf/bitbake.conf
> >> +++ b/meta/conf/bitbake.conf
> >> @@ -833,7 +833,7 @@ XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT}
> >> --threads=${XZ_THREADS}"
> >>   XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"
> >>   # Default parallelism for zstd
> >> -ZSTD_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
> >> +ZSTD_THREADS ?= "${@int(oe.utils.cpu_count(at_least=4)/2)}"
> >
> > Then why not just limit it for the large setups you are referring to in
> the
> > example, for instance like
> >
> > ZSTD_THREADS ?= "${@min(int(oe.utils.cpu_count(at_least=4)),  sane
> > value of your choice>)}"
>
> This sounds like a good choice.
>
> >
> > BTW this can be also done in your local.conf - as Alex already said,
> there is
>
> Set it in my local.conf can make it work, but it isn't good for oe-core's
> OOBE
> since the issue still exists.
>
> > simply no reason to make it slower for everyone, while also half the
> threads of
>
> The whole build contains a lot of tasks, for lower cpu cores, the total
> resources are fixed, so when zstd uses less mem, other tasks can use more.
>
> Limit zstd's thread doesn't make the build slower, but faster in my
> testing,
> it would be great if you can help test it. The simple testing commands are:
>
> # Make everything ready:
> $ bitbake linux-yocto
>
> # Before the patch, set ZSTD_THREADS to 128 and run 6 times:
> $ for i in `seq 0 5`; do time bitbake linux-yocto -cpackage_write_rpm -f >
> before_$i.log; done
>
> Note, the time will be printed on the screen, not the before.log, the
> before.log
> is for strip the logs and then we can read the time easier.
>
> What I get is:
> real2m12.079s
> real2m0.177s
> real1m52.426s
> real2m3.396s
> real2m16.018s
> real1m58.595s
>
> Drop the first build time since it *may* contain parsing time, and for the
> last
> five builds:
>
> Total: 609 seconds
> Average: 609 / 5.0 = 121.8
>
>
> # After the patch, set ZSTD_THREADS to 64, and run 6 times:
> $ for i in `seq 0 5`; do time bitbake linux-yocto -cpackage_write_rpm -f >
> after_$i.log; done
>
> What I get is:
> real1m50.017s
> real1m50.400s
> real1m53.174s
> real2m4.817s
> real1m53.476s
> real1m56.794s
>
> Drop the first build time since it *may* contain parsing time, and for the
> last
> five builds:
>
> Total: 576 seconds
> Average: 576 / 5.0 = 115.2
>
> So the smaller number is faster than the larger one.
>
> // Robert
>
> > a 128 core machine could be a trouble some setup.
> >
> > Last time I had this issue (with XZ back then) I used
> > "${@min(int(oe.utils.cpu_count(at_least=4)), 20}"
> >
> >>   ZSTD_THREADS[vardepvalue] = "1"
> >>   # Limit the number of threads that OpenMP libraries will use.
> Otherwise they
> >>
> >>
> >>
> >>
> >>
>
> 
>
>

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#158005): 
https://lists.openembedded.org/g/openembedded-core/message/158005
Mute This Topic: https://lists.openembedded.org/mt/86926962/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH 1/1] bitbake.conf: Set ZSTD_THREADS to half of cpu number

[Robert Yang]

On 11/9/21 4:41 PM, Konrad Weihmann wrote:



On 09.11.21 09:48, Robert Yang wrote:

The original value is very easy to cause do_packge error when cpu number is
larger, for example, 128 cores and 512G mem:

error: create archive failed: cpio: write failed - Cannot allocate memory"

Set the ZSTD_THREADS to half of the CPU number can avoid the error in my
testing.

Signed-off-by: Robert Yang 
---
  meta/conf/bitbake.conf | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 71c1e52ad6..46ebf5113f 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -833,7 +833,7 @@ XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} 
--threads=${XZ_THREADS}"

  XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"
  # Default parallelism for zstd
-ZSTD_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
+ZSTD_THREADS ?= "${@int(oe.utils.cpu_count(at_least=4)/2)}"


Then why not just limit it for the large setups you are referring to in the 
example, for instance like


ZSTD_THREADS ?= "${@min(int(oe.utils.cpu_count(at_least=4)), value of your choice>)}"


This sounds like a good choice.



BTW this can be also done in your local.conf - as Alex already said, there is 


Set it in my local.conf can make it work, but it isn't good for oe-core's OOBE
since the issue still exists.

simply no reason to make it slower for everyone, while also half the threads of 


The whole build contains a lot of tasks, for lower cpu cores, the total
resources are fixed, so when zstd uses less mem, other tasks can use more.

Limit zstd's thread doesn't make the build slower, but faster in my testing,
it would be great if you can help test it. The simple testing commands are:

# Make everything ready:
$ bitbake linux-yocto

# Before the patch, set ZSTD_THREADS to 128 and run 6 times:
$ for i in `seq 0 5`; do time bitbake linux-yocto -cpackage_write_rpm -f > 
before_$i.log; done


Note, the time will be printed on the screen, not the before.log, the before.log
is for strip the logs and then we can read the time easier.

What I get is:
real2m12.079s
real2m0.177s
real1m52.426s
real2m3.396s
real2m16.018s
real1m58.595s

Drop the first build time since it *may* contain parsing time, and for the last
five builds:

Total: 609 seconds
Average: 609 / 5.0 = 121.8


# After the patch, set ZSTD_THREADS to 64, and run 6 times:
$ for i in `seq 0 5`; do time bitbake linux-yocto -cpackage_write_rpm -f > 
after_$i.log; done


What I get is:
real1m50.017s
real1m50.400s
real1m53.174s
real2m4.817s
real1m53.476s
real1m56.794s

Drop the first build time since it *may* contain parsing time, and for the last
five builds:

Total: 576 seconds
Average: 576 / 5.0 = 115.2

So the smaller number is faster than the larger one.

// Robert


a 128 core machine could be a trouble some setup.

Last time I had this issue (with XZ back then) I used 
"${@min(int(oe.utils.cpu_count(at_least=4)), 20}"



  ZSTD_THREADS[vardepvalue] = "1"
  # Limit the number of threads that OpenMP libraries will use. Otherwise they






-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#158004): 
https://lists.openembedded.org/g/openembedded-core/message/158004
Mute This Topic: https://lists.openembedded.org/mt/86926962/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH 1/1] bitbake.conf: Set ZSTD_THREADS to half of cpu number

[Jose Quaresma]

Robert Yang  escreveu no dia terça, 9/11/2021
à(s) 08:12:

> The original value is very easy to cause do_packge error when cpu number is
> larger, for example, 128 cores and 512G mem:
>
> error: create archive failed: cpio: write failed - Cannot allocate memory"
>
> Set the ZSTD_THREADS to half of the CPU number can avoid the error in my
> testing.
>
> Signed-off-by: Robert Yang 
> ---
>  meta/conf/bitbake.conf | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index 71c1e52ad6..46ebf5113f 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -833,7 +833,7 @@ XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT}
> --threads=${XZ_THREADS}"
>  XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"
>
>  # Default parallelism for zstd
> -ZSTD_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
> +ZSTD_THREADS ?= "${@int(oe.utils.cpu_count(at_least=4)/2)}"
>

This will slow down all the systems with lower cpu cores.


>  ZSTD_THREADS[vardepvalue] = "1"
>
>  # Limit the number of threads that OpenMP libraries will use. Otherwise
> they
> --
> 2.17.1
>
>
> 
>
>

-- 
Best regards,

José Quaresma

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#158002): 
https://lists.openembedded.org/g/openembedded-core/message/158002
Mute This Topic: https://lists.openembedded.org/mt/86926962/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH 1/1] bitbake.conf: Set ZSTD_THREADS to half of cpu number

[Konrad Weihmann]

On 09.11.21 09:48, Robert Yang wrote:

The original value is very easy to cause do_packge error when cpu number is
larger, for example, 128 cores and 512G mem:

error: create archive failed: cpio: write failed - Cannot allocate memory"

Set the ZSTD_THREADS to half of the CPU number can avoid the error in my
testing.

Signed-off-by: Robert Yang 
---
  meta/conf/bitbake.conf | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 71c1e52ad6..46ebf5113f 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -833,7 +833,7 @@ XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} 
--threads=${XZ_THREADS}"
  XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"
  
  # Default parallelism for zstd

-ZSTD_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
+ZSTD_THREADS ?= "${@int(oe.utils.cpu_count(at_least=4)/2)}"


Then why not just limit it for the large setups you are referring to in 
the example, for instance like


ZSTD_THREADS ?= "${@min(int(oe.utils.cpu_count(at_least=4)), sane value of your choice>)}"


BTW this can be also done in your local.conf - as Alex already said, 
there is simply no reason to make it slower for everyone, while also 
half the threads of a 128 core machine could be a trouble some setup.


Last time I had this issue (with XZ back then) I used 
"${@min(int(oe.utils.cpu_count(at_least=4)), 20}"



  ZSTD_THREADS[vardepvalue] = "1"
  
  # Limit the number of threads that OpenMP libraries will use. Otherwise they







-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#158001): 
https://lists.openembedded.org/g/openembedded-core/message/158001
Mute This Topic: https://lists.openembedded.org/mt/86926962/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH 1/1] bitbake.conf: Set ZSTD_THREADS to half of cpu number

[Alexander Kanavin]

But does this mean we need to globally restrict this and make things twice
slower for everyone? I'm not sure.


Alex

On Tue, 9 Nov 2021 at 09:12, Robert Yang  wrote:

> The original value is very easy to cause do_packge error when cpu number is
> larger, for example, 128 cores and 512G mem:
>
> error: create archive failed: cpio: write failed - Cannot allocate memory"
>
> Set the ZSTD_THREADS to half of the CPU number can avoid the error in my
> testing.
>
> Signed-off-by: Robert Yang 
> ---
>  meta/conf/bitbake.conf | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index 71c1e52ad6..46ebf5113f 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -833,7 +833,7 @@ XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT}
> --threads=${XZ_THREADS}"
>  XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"
>
>  # Default parallelism for zstd
> -ZSTD_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
> +ZSTD_THREADS ?= "${@int(oe.utils.cpu_count(at_least=4)/2)}"
>  ZSTD_THREADS[vardepvalue] = "1"
>
>  # Limit the number of threads that OpenMP libraries will use. Otherwise
> they
> --
> 2.17.1
>
>
> 
>
>

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#158000): 
https://lists.openembedded.org/g/openembedded-core/message/158000
Mute This Topic: https://lists.openembedded.org/mt/86926962/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH 1/1] bitbake.conf: Set ZSTD_THREADS to half of cpu number

[Robert Yang]

The original value is very easy to cause do_packge error when cpu number is
larger, for example, 128 cores and 512G mem:

error: create archive failed: cpio: write failed - Cannot allocate memory"

Set the ZSTD_THREADS to half of the CPU number can avoid the error in my
testing.

Signed-off-by: Robert Yang 
---
 meta/conf/bitbake.conf | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 71c1e52ad6..46ebf5113f 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -833,7 +833,7 @@ XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} 
--threads=${XZ_THREADS}"
 XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"
 
 # Default parallelism for zstd
-ZSTD_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
+ZSTD_THREADS ?= "${@int(oe.utils.cpu_count(at_least=4)/2)}"
 ZSTD_THREADS[vardepvalue] = "1"
 
 # Limit the number of threads that OpenMP libraries will use. Otherwise they
-- 
2.17.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#157998): 
https://lists.openembedded.org/g/openembedded-core/message/157998
Mute This Topic: https://lists.openembedded.org/mt/86926962/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-