Re: [OE-core] [RFC PATCH] populate_sdk_base: add support for zstd compression (and default to it)

2021-10-17 Thread Alexander Kanavin 
On Sat, 16 Oct 2021 at 21:21, Joshua Watt  wrote:

>
> Is this the maximum compression level for zstd? Also, are any of these
> using parallel compression?
>

Yes, they all run with the same standard cpu cores setting from
bitbake.conf. xz additionally has a RAM cap.

zstd has further compression levels, but I didn't try them - you're weclome
to:
--ultra : enable levels beyond 19, up to 22 (requires more memory)

Alex

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#157033): 
https://lists.openembedded.org/g/openembedded-core/message/157033
Mute This Topic: https://lists.openembedded.org/mt/86353954/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [RFC PATCH] populate_sdk_base: add support for zstd compression (and default to it)

2021-10-16 Thread Joshua Watt 
On Fri, Oct 15, 2021, 3:55 PM Alexander Kanavin 
wrote:

> On Fri, 15 Oct 2021 at 19:29, Richard Purdie <
> richard.pur...@linuxfoundation.org> wrote:
>
>>
>> Can you quantify "more"?
>>
>> I'd be interested to know if a higher compression helps and what the
>> time/cost
>> tradeoff is too. Not sure I'm keep to have it increase in size. This
>> likely
>> increases the eSDK size a lot too?
>>
>
> The benefit is indeed not unquestionable, hence the RFC.
>
> 1. The eSDK is mostly made of already compressed sstate artifacts, so the
> effect on size is minimal and the zstd's time benefit is clear:
>
> $ bitbake -c populate_sdk_ext core-image-sato (to have everything ready)
> $ bitbake -c cleansstate core-image-sato buildtools-tarball
> $ time bitbake -c populate_sdk_ext core-image-sato
>
> xz time/size
> 4m46s 1.7G
>
> zstd compression level/time/size
> 19 3m4s 1.7G
> 10 2m44s 1.8G
>
> 2. For the basic SDK (bitbake commands are similar), xz's better (but
> slower) compression does show clearly, but zstd's time wins are similar :
>
> xz time/size
> 5m20s 581M
>
> zstd compression level/time/size:
> 19 4m5s 837M
> 15 3m44s 979M
> 10 3m22s 993M
> default(3) 3m19s 1.1G
>

Is this the maximum compression level for zstd? Also, are any of these
using parallel compression?



> Alex
>
> 
>
>

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#157018): 
https://lists.openembedded.org/g/openembedded-core/message/157018
Mute This Topic: https://lists.openembedded.org/mt/86353954/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [RFC PATCH] populate_sdk_base: add support for zstd compression (and default to it)

2021-10-16 Thread Richard Purdie 
On Fri, 2021-10-15 at 22:54 +0200, Alexander Kanavin wrote:
> On Fri, 15 Oct 2021 at 19:29, Richard Purdie
>  wrote:
> > 
> > Can you quantify "more"?
> > 
> > I'd be interested to know if a higher compression helps and what the
> > time/cost
> > tradeoff is too. Not sure I'm keep to have it increase in size. This likely
> > increases the eSDK size a lot too?
> > 
> 
> 
> The benefit is indeed not unquestionable, hence the RFC.

Thanks, I think we need data to be able to comment effectively! :)

> 1. The eSDK is mostly made of already compressed sstate artifacts, so the
> effect on size is minimal and the zstd's time benefit is clear:
> 
> $ bitbake -c populate_sdk_ext core-image-sato (to have everything ready)
> $ bitbake -c cleansstate core-image-sato buildtools-tarball
> $ time bitbake -c populate_sdk_ext core-image-sato
> 
> xz time/size
> 4m46s 1.7G
> 
> zstd compression level/time/size
> 19 3m4s 1.7G
> 10 2m44s 1.8G

For the eSDK this looks like an ok change.

> 2. For the basic SDK (bitbake commands are similar), xz's better (but slower)
> compression does show clearly, but zstd's time wins are similar : 
> 
> xz time/size
> 5m20s 581M
> 
> zstd compression level/time/size:
> 19 4m5s 837M
> 15 3m44s 979M
> 10 3m22s 993M
> default(3) 3m19s 1.1G

but nearly doubling the size of the standard SDK isn't going to be well received
by people using it, those numbers don't look like they can work even if it is
faster. The key things are primarily size and secondly decompression time,
compression time is in third place and unfortunate but something the build
systems can afford to pay.

Cheers,

Richard






-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#157012): 
https://lists.openembedded.org/g/openembedded-core/message/157012
Mute This Topic: https://lists.openembedded.org/mt/86353954/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [RFC PATCH] populate_sdk_base: add support for zstd compression (and default to it)

2021-10-15 Thread Alexander Kanavin 
On Fri, 15 Oct 2021 at 19:29, Richard Purdie <
richard.pur...@linuxfoundation.org> wrote:

>
> Can you quantify "more"?
>
> I'd be interested to know if a higher compression helps and what the
> time/cost
> tradeoff is too. Not sure I'm keep to have it increase in size. This likely
> increases the eSDK size a lot too?
>

The benefit is indeed not unquestionable, hence the RFC.

1. The eSDK is mostly made of already compressed sstate artifacts, so the
effect on size is minimal and the zstd's time benefit is clear:

$ bitbake -c populate_sdk_ext core-image-sato (to have everything ready)
$ bitbake -c cleansstate core-image-sato buildtools-tarball
$ time bitbake -c populate_sdk_ext core-image-sato

xz time/size
4m46s 1.7G

zstd compression level/time/size
19 3m4s 1.7G
10 2m44s 1.8G

2. For the basic SDK (bitbake commands are similar), xz's better (but
slower) compression does show clearly, but zstd's time wins are similar :

xz time/size
5m20s 581M

zstd compression level/time/size:
19 4m5s 837M
15 3m44s 979M
10 3m22s 993M
default(3) 3m19s 1.1G

Alex

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#157009): 
https://lists.openembedded.org/g/openembedded-core/message/157009
Mute This Topic: https://lists.openembedded.org/mt/86353954/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [RFC PATCH] populate_sdk_base: add support for zstd compression (and default to it)

2021-10-15 Thread Khem Raj 



On 10/15/21 10:29 AM, Richard Purdie wrote:

On Fri, 2021-10-15 at 19:06 +0200, Alexander Kanavin wrote:

This allows significantly faster do_populate_sdk() (5 min -> 3 min for
the sato sdk on my machine) and decompression, at the cost of SDK file
taking up more disk space.


Can you quantify "more"?

I'd be interested to know if a higher compression helps and what the time/cost
tradeoff is too. Not sure I'm keep to have it increase in size. This likely
increases the eSDK size a lot too?


since eSDKs are generate once use many times compression time is not as 
important, decompression time is important, however, they are also 
transmitted via network which means we might lose out the decompression 
gain towards network transfer time. Perhaps it will be good to get a 
wholesome picture.





If that's a significant concern, it's possible to increase the zstd
compression level or even go back to xz compression.

Signed-off-by: Alexander Kanavin 
---
  meta/classes/populate_sdk_base.bbclass | 10 --
  meta/files/toolchain-shar-extract.sh   |  9 -
  2 files changed, 12 insertions(+), 7 deletions(-)

diff --git a/meta/classes/populate_sdk_base.bbclass 
b/meta/classes/populate_sdk_base.bbclass
index 49e166e697..92a23e2325 100644
--- a/meta/classes/populate_sdk_base.bbclass
+++ b/meta/classes/populate_sdk_base.bbclass
@@ -50,17 +50,23 @@ TOOLCHAIN_TARGET_TASK_ATTEMPTONLY ?= ""
  TOOLCHAIN_OUTPUTNAME ?= "${SDK_NAME}-toolchain-${SDK_VERSION}"
  
  # Default archived SDK's suffix

-SDK_ARCHIVE_TYPE ?= "tar.xz"
+SDK_ARCHIVE_TYPE ?= "tar.zst"
  SDK_XZ_COMPRESSION_LEVEL ?= "-9"
  SDK_XZ_OPTIONS ?= "${XZ_DEFAULTS} ${SDK_XZ_COMPRESSION_LEVEL}"
  
-# To support different sdk type according to SDK_ARCHIVE_TYPE, now support zip and tar.xz

+SDK_ZSTD_COMPRESSION_LEVEL ?= ""
+SDK_ZSTD_OPTIONS ?= "-T${ZSTD_THREADS} ${SDK_ZSTD_COMPRESSION_LEVEL}"
+
+# To support different sdk type according to SDK_ARCHIVE_TYPE, now support 
zip, tar.zst and tar.xz
  python () {
  if d.getVar('SDK_ARCHIVE_TYPE') == 'zip':
 d.setVar('SDK_ARCHIVE_DEPENDS', 'zip-native')
 # SDK_ARCHIVE_CMD used to generate archived sdk 
${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE} from input dir 
${SDK_OUTPUT}/${SDKPATH} to output dir ${SDKDEPLOYDIR}
 # recommand to cd into input dir first to avoid archive with buildpath
 d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; zip -r -y 
${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE} .')
+elif d.getVar('SDK_ARCHIVE_TYPE') == 'tar.zst':
+   d.setVar('SDK_ARCHIVE_DEPENDS', 'zstd-native')
+   d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; tar 
${SDKTAROPTS} -cf - . | zstd ${SDK_ZSTD_OPTIONS} > 
${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE}')
  else:
 d.setVar('SDK_ARCHIVE_DEPENDS', 'xz-native')
 d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; tar 
${SDKTAROPTS} -cf - . | xz ${SDK_XZ_OPTIONS} > 
${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE}')
diff --git a/meta/files/toolchain-shar-extract.sh 
b/meta/files/toolchain-shar-extract.sh
index 4386b985bb..4b8892af79 100644
--- a/meta/files/toolchain-shar-extract.sh
+++ b/meta/files/toolchain-shar-extract.sh
@@ -51,11 +51,6 @@ if [ "$INST_ARCH" != "$SDK_ARCH" ]; then
fi
  fi
  
-if ! xz -V > /dev/null 2>&1; then

-   echo "Error: xz is required for installation of this SDK, please install it 
first"
-   exit 1
-fi
-




Deleting that is fine for testing but I think the final patch needs to do
better! ;-)

Cheers,

Richard






-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#157007): 
https://lists.openembedded.org/g/openembedded-core/message/157007
Mute This Topic: https://lists.openembedded.org/mt/86353954/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [RFC PATCH] populate_sdk_base: add support for zstd compression (and default to it)

2021-10-15 Thread Richard Purdie 
On Fri, 2021-10-15 at 19:06 +0200, Alexander Kanavin wrote:
> This allows significantly faster do_populate_sdk() (5 min -> 3 min for
> the sato sdk on my machine) and decompression, at the cost of SDK file
> taking up more disk space.

Can you quantify "more"?

I'd be interested to know if a higher compression helps and what the time/cost
tradeoff is too. Not sure I'm keep to have it increase in size. This likely
increases the eSDK size a lot too?

> If that's a significant concern, it's possible to increase the zstd
> compression level or even go back to xz compression.
> 
> Signed-off-by: Alexander Kanavin 
> ---
>  meta/classes/populate_sdk_base.bbclass | 10 --
>  meta/files/toolchain-shar-extract.sh   |  9 -
>  2 files changed, 12 insertions(+), 7 deletions(-)
> 
> diff --git a/meta/classes/populate_sdk_base.bbclass 
> b/meta/classes/populate_sdk_base.bbclass
> index 49e166e697..92a23e2325 100644
> --- a/meta/classes/populate_sdk_base.bbclass
> +++ b/meta/classes/populate_sdk_base.bbclass
> @@ -50,17 +50,23 @@ TOOLCHAIN_TARGET_TASK_ATTEMPTONLY ?= ""
>  TOOLCHAIN_OUTPUTNAME ?= "${SDK_NAME}-toolchain-${SDK_VERSION}"
>  
>  # Default archived SDK's suffix
> -SDK_ARCHIVE_TYPE ?= "tar.xz"
> +SDK_ARCHIVE_TYPE ?= "tar.zst"
>  SDK_XZ_COMPRESSION_LEVEL ?= "-9"
>  SDK_XZ_OPTIONS ?= "${XZ_DEFAULTS} ${SDK_XZ_COMPRESSION_LEVEL}"
>  
> -# To support different sdk type according to SDK_ARCHIVE_TYPE, now support 
> zip and tar.xz
> +SDK_ZSTD_COMPRESSION_LEVEL ?= ""
> +SDK_ZSTD_OPTIONS ?= "-T${ZSTD_THREADS} ${SDK_ZSTD_COMPRESSION_LEVEL}"
> +
> +# To support different sdk type according to SDK_ARCHIVE_TYPE, now support 
> zip, tar.zst and tar.xz
>  python () {
>  if d.getVar('SDK_ARCHIVE_TYPE') == 'zip':
> d.setVar('SDK_ARCHIVE_DEPENDS', 'zip-native')
> # SDK_ARCHIVE_CMD used to generate archived sdk 
> ${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE} from input dir 
> ${SDK_OUTPUT}/${SDKPATH} to output dir ${SDKDEPLOYDIR}
> # recommand to cd into input dir first to avoid archive with buildpath
> d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; zip -r -y 
> ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE} .')
> +elif d.getVar('SDK_ARCHIVE_TYPE') == 'tar.zst':
> +   d.setVar('SDK_ARCHIVE_DEPENDS', 'zstd-native')
> +   d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; tar 
> ${SDKTAROPTS} -cf - . | zstd ${SDK_ZSTD_OPTIONS} > 
> ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE}')
>  else:
> d.setVar('SDK_ARCHIVE_DEPENDS', 'xz-native')
> d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; tar 
> ${SDKTAROPTS} -cf - . | xz ${SDK_XZ_OPTIONS} > 
> ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE}')
> diff --git a/meta/files/toolchain-shar-extract.sh 
> b/meta/files/toolchain-shar-extract.sh
> index 4386b985bb..4b8892af79 100644
> --- a/meta/files/toolchain-shar-extract.sh
> +++ b/meta/files/toolchain-shar-extract.sh
> @@ -51,11 +51,6 @@ if [ "$INST_ARCH" != "$SDK_ARCH" ]; then
>   fi
>  fi
>  
> -if ! xz -V > /dev/null 2>&1; then
> - echo "Error: xz is required for installation of this SDK, please 
> install it first"
> - exit 1
> -fi
> -
> 
> 

Deleting that is fine for testing but I think the final patch needs to do
better! ;-)

Cheers,

Richard


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#157005): 
https://lists.openembedded.org/g/openembedded-core/message/157005
Mute This Topic: https://lists.openembedded.org/mt/86353954/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-