On Tue, 2020-08-11 at 16:09 +0800, Yu, Mingli wrote:
> Hi RP,
>
> On 8/11/20 3:57 PM, Richard Purdie wrote:
> > On Tue, 2020-08-11 at 15:22 +0800, Yu, Mingli wrote:
> > > From: Mingli Yu <[email protected]>
> > >
> > > pigz, which stands for parallel implementation of gzip, is a
> > > fully functional replacement for gzip that exploits multiple
> > > processors and multiple cores to the hilt when compressing data.
> > >
> > > do_package uses pigz to compress data to speed up the time
> > > if there is pigz available on the build server.
> > >
> > > But for some big package such as clang, there comes below
> > > error during do_package.
> > > gzip: stdout: Cannot allocate memory
> > > tar: TOPDIR/sstate-cache/20/80/sstate:clang:cortexa57-wrs-
> > > linux:10.0.1:r0:cortexa57:3:20808f8a746b00ec9470a81057c8595d878cf
> > > e8c565cfbcbbaa3ea870528ee34_package.tgz.QF5dss8M: Wrote only 8192
> > > of 10240 bytes
> > > tar: Child returned status 1
> > > tar: Error is not recoverable: exiting now
> > >
> > > So set the default pigz thread and the user also can customize
> > > the pigz thread such as via PIGZ = "-p 3" or PIGZ_pn-clang = "-p
> > > 3"
> > > for the individual package in local.conf to avoid eating so much
> > > memory.
> > >
> > > Signed-off-by: Mingli Yu <[email protected]>
> > > ---
> > > meta/conf/bitbake.conf | 3 +++
> > > 1 file changed, 3 insertions(+)
> > >
> > > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> > > index 353caacef9..bc87e78d78 100644
> > > --- a/meta/conf/bitbake.conf
> > > +++ b/meta/conf/bitbake.conf
> > > @@ -804,6 +804,9 @@ XZ_THREADS ?= "${@oe.utils.cpu_count(at_least
> > > =2)}"
> > > XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} --
> > > threads=${XZ_THREADS}"
> > > XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"
> > >
> > > +# Default parallelism and resource usage for pigz
> > > +export PIGZ = "-p ${@oe.utils.cpu_count(at_least=2)}"
> >
> > Why not just set this in the cases you need it in and leave the
> > defaults alone? You can just export it in the tasks which need it
> > too.
>
> Actually I just want to use this logic in this patch altogether
> PIGZ_pn-clang = "-p 3" in local.conf to limit the pigz thread number
> for clang.
>
> Without this patch, I think PIGZ_pn-clang = "-p 3" doesn't take
> effect.
You'll have to use something like:
PIGZEXPORT = "0"
PIGZEXPORT_pn-clang = "1"
PIGZ[export] = "${PIGZEXPORT}"
> > This change actually makes the sstate number of cpus specific which
> > we
> > definitely don't want.
>
> From my understanding, the default behavior for pigz is to take all
> online cores, it should be the same with export PIGZ = "-p
> ${@oe.utils.cpu_count(at_least=2)}".
I understand why you want to do it and what it does. It breaks sstate
signatures for everyone else though so it is not acceptable.
Cheers,
Richard
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#141367):
https://lists.openembedded.org/g/openembedded-core/message/141367
Mute This Topic: https://lists.openembedded.org/mt/76121921/21656
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
