On 9/13/22 15:20, Alex Feinman wrote:
I do have some numbers. When I was selling this change internally, I
did a comparison on our internal build.
Combined write IPK times (Σ t do_package_write_ipk)
xz 162m 35s
gz 52m 13s
zstd 33m 49s
Compression rate for zstd was closer to xz than to gz but not as good
as xz. For systems that have to cache packages on the device with
limited storage xz might be a better option, but for the bulk of
projects zstd is the best choice
Additionally, zstd offers much faster decompression than xz so the
rootfs build step that includes unpacking all of the ipks, takes 3m
58s with xz and 2m 44s with zstd.
One other thing of note - if your build includes debug packages, some
may be quite large. E.g. one of our components produces a 2.2 GB debug
package (uncompressed). On large files xz requires a disproportionally
large amount of time resulting in 15 minutes needed to simply write
ipk for the abovementioned packages, whereas zstd took about 45 sec.
For frequent tasks like bitbaking a single package this translates in
a lot of saved time.
Those are certainly compelling performance improvements. Assuming that
the final data-segment size is within 5%-ish of xz, then I would agree
with the rest of the thread that it should probably be the contemporary
default.
And if we make it the default compressor for OE IPKs, then obviously my
criticism in the original PR is satisfied.
Bottom line - I think making xz a default package compressor was not
entirely thought through. gzip or zstd is what the default should be.
ZStandard support was only added to opkg last September [1]. Before
that, xz was the new hotness that replaced gzip. :)
[1]
https://git.yoctoproject.org/opkg/commit/?id=5dead419e94bce2e6b743ad786c1daec0e1aa294
One final note: I could not find a reasonable explanation for why
opkg-tools require code changes to support a different compressor. BSD
tar and GNU tar both can easily accept compressors that they have no
idea about (via -I option) because all of them provide a unified
command line interface for use in pipes. If this were done similar to
tar, we could have used any compressor we wanted, including the
multithreaded versions (zstdmt)
Well, presumably IPK creation tools can only support the matrix of
compression algorithms which your opkg binary can decompress. I suppose
someone could try to implement a plugable compression module system for
opkg. But given that nearly everyone uses opkg in an embedded context,
I'm not sure it would get much use.
On Tue, Sep 13, 2022 at 12:43 PM Khem Raj <raj.k...@gmail.com> wrote:
On Tue, Sep 13, 2022 at 12:19 PM Alex Stewart
<alex.stew...@ni.com> wrote:
>
> ACK from me - apart from enabling zstd by default.
>
> On 9/13/22 07:37, Etienne Cordonnier via lists.openembedded.org
<https://urldefense.com/v3/__http://lists.openembedded.org__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_bxrNjvo$>
wrote:
> > This allows the use of zstd for opkg packages by using
OPKGBUILDCMD:
> > OPKGBUILDCMD = "opkg-build -Z zstd"
> >
> > Signed-off-by: Alex Feinman <afein...@snap.com>
> > Signed-off-by: Etienne Cordonnier <ecordonn...@snap.com>
> > ---
> > meta/recipes-devtools/opkg/opkg_0.6.0.bb
<https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
| 3 ++-
> > 1 file changed, 2 insertions(+), 1 deletion(-)
> >
> > diff --git a/meta/recipes-devtools/opkg/opkg_0.6.0.bb
<https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
<https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
> > index 7b351e8123..e38d9d6f3f 100644
> > --- a/meta/recipes-devtools/opkg/opkg_0.6.0.bb
<https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
> > +++ b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
<https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
> > @@ -30,7 +30,7 @@ inherit autotools pkgconfig ptest
> > target_localstatedir := "${localstatedir}"
> > OPKGLIBDIR ??= "${target_localstatedir}/lib"
> >
> > -PACKAGECONFIG ??= "libsolv"
> > +PACKAGECONFIG ??= "libsolv zstd"
>
> Building in zstd support by default is a little suspect to me.
>
> Unless I'm mistaken, OE-core will only build xz-compressed IPKs by
> default. So zstd support would be unnecessary for a distro
integrator
> who just uses upstream OE-core.
>
> For distros which use zstd compression in their packages, I think it
> would be more appropriate to overwrite the opkg PACKAGECONFIG in a
> .bbappend.
>
This is perhaps fine. I do wonder if there is some performance
comparison data between xz and zstd compressed ipks
with opkg, it might help users on making this choice and also if we
should consider using
zstd by default at some point or not.
> Is there something I'm not considering here?
>
> >
> > PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\
> > gnupg gpgme libgpg-error,\
> > @@ -39,6 +39,7 @@ PACKAGECONFIG[gpg] =
"--enable-gpg,--disable-gpg,\
> > PACKAGECONFIG[curl] = "--enable-curl,--disable-curl,curl"
> > PACKAGECONFIG[ssl-curl] =
"--enable-ssl-curl,--disable-ssl-curl,curl openssl"
> > PACKAGECONFIG[sha256] = "--enable-sha256,--disable-sha256"
> > +PACKAGECONFIG[zstd] = "--enable-zstd,--disable-zstd,zstd"
> > PACKAGECONFIG[libsolv] =
"--with-libsolv,--without-libsolv,libsolv"
> >
> > EXTRA_OECONF:class-native =
"--localstatedir=/${@os.path.relpath('${localstatedir}',
'${STAGING_DIR_NATIVE}')}
--sysconfdir=/${@os.path.relpath('${sysconfdir}',
'${STAGING_DIR_NATIVE}')}"
> >
> >
> >
>
> --
> Alex Stewart
> Software Engineer - NI Real-Time OS
> NI (National Instruments)
>
> alex.stew...@ni.com
>
>
>
>
--
Alex Stewart
Software Engineer - NI Real-Time OS
NI (National Instruments)
alex.stew...@ni.com
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170613):
https://lists.openembedded.org/g/openembedded-core/message/170613
Mute This Topic: https://lists.openembedded.org/mt/93654146/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
