Re: [OE-core] [PATCH] populate_sdk_base: use xz -T instead of pixz

[Burton, Ross]

On 9 April 2018 at 22:20, Khem Raj  wrote:
> On Mon, Apr 9, 2018 at 1:44 PM, Burton, Ross  wrote:
>> On 9 April 2018 at 21:13, Andre McCurdy  wrote:
>>> On Mon, Apr 9, 2018 at 8:19 AM, Ross Burton  wrote:
 xz has native support for threaded compression now and SDK creation was 
 the only
 part of oe-core which is using pixz instead of xz.

 Not only does this remove pixz-native from the SDK dependencies, but in my
 limited testing xz -T0 is slightly faster and produces smaller archives 
 than
 pixz for the same input.

 Signed-off-by: Ross Burton 
 ---

  # We want the MULTIARCH_TARGET_SYS to point to the TUNE_PKGARCH, not 
 PACKAGE_ARCH as it
 @@ -225,7 +225,7 @@ fakeroot tar_sdk() {
 # Package it up
 mkdir -p ${SDKDEPLOYDIR}
 cd ${SDK_OUTPUT}/${SDKPATH}
 -   tar ${SDKTAROPTS} -cf - . | pixz > 
 ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
 +   tar ${SDKTAROPTS} -cf - . | xz -T 0 > 
 ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
>>>
>>> Since -T 0 will use all available CPUs (ie any attempts the user may
>>> have made to limit parallelism via BB_NUMBER_THREADS or PARALLEL_MAKE
>>> will be ignored), perhaps it's worth adding something like
>>> "--memlimit=70%" to try to give some protection for environments with
>>> lots of CPUs but not so much DRAM?
>>
>> There's a few places where -T0 is passed already: tar.xz image
>> creation and opkg creation, so I guess we need to centralise this
>> somewhere too.
>>
>
> May be use XZ_OPT env bariable and param to -T could use the bitbake
> calculated number of CPUs

-T0 is use as many threads as there are cores, which is the same as
the bitbake function.  Re-using BB_NUMBER_THREADS might work.

Ross
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] populate_sdk_base: use xz -T instead of pixz

[Khem Raj]

On Mon, Apr 9, 2018 at 1:44 PM, Burton, Ross  wrote:
> On 9 April 2018 at 21:13, Andre McCurdy  wrote:
>> On Mon, Apr 9, 2018 at 8:19 AM, Ross Burton  wrote:
>>> xz has native support for threaded compression now and SDK creation was the 
>>> only
>>> part of oe-core which is using pixz instead of xz.
>>>
>>> Not only does this remove pixz-native from the SDK dependencies, but in my
>>> limited testing xz -T0 is slightly faster and produces smaller archives than
>>> pixz for the same input.
>>>
>>> Signed-off-by: Ross Burton 
>>> ---
>>>
>>>  # We want the MULTIARCH_TARGET_SYS to point to the TUNE_PKGARCH, not 
>>> PACKAGE_ARCH as it
>>> @@ -225,7 +225,7 @@ fakeroot tar_sdk() {
>>> # Package it up
>>> mkdir -p ${SDKDEPLOYDIR}
>>> cd ${SDK_OUTPUT}/${SDKPATH}
>>> -   tar ${SDKTAROPTS} -cf - . | pixz > 
>>> ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
>>> +   tar ${SDKTAROPTS} -cf - . | xz -T 0 > 
>>> ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
>>
>> Since -T 0 will use all available CPUs (ie any attempts the user may
>> have made to limit parallelism via BB_NUMBER_THREADS or PARALLEL_MAKE
>> will be ignored), perhaps it's worth adding something like
>> "--memlimit=70%" to try to give some protection for environments with
>> lots of CPUs but not so much DRAM?
>
> There's a few places where -T0 is passed already: tar.xz image
> creation and opkg creation, so I guess we need to centralise this
> somewhere too.
>

May be use XZ_OPT env bariable and param to -T could use the bitbake
calculated number of CPUs

> Ross
> --
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] populate_sdk_base: use xz -T instead of pixz

[Burton, Ross]

On 9 April 2018 at 21:13, Andre McCurdy  wrote:
> On Mon, Apr 9, 2018 at 8:19 AM, Ross Burton  wrote:
>> xz has native support for threaded compression now and SDK creation was the 
>> only
>> part of oe-core which is using pixz instead of xz.
>>
>> Not only does this remove pixz-native from the SDK dependencies, but in my
>> limited testing xz -T0 is slightly faster and produces smaller archives than
>> pixz for the same input.
>>
>> Signed-off-by: Ross Burton 
>> ---
>>
>>  # We want the MULTIARCH_TARGET_SYS to point to the TUNE_PKGARCH, not 
>> PACKAGE_ARCH as it
>> @@ -225,7 +225,7 @@ fakeroot tar_sdk() {
>> # Package it up
>> mkdir -p ${SDKDEPLOYDIR}
>> cd ${SDK_OUTPUT}/${SDKPATH}
>> -   tar ${SDKTAROPTS} -cf - . | pixz > 
>> ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
>> +   tar ${SDKTAROPTS} -cf - . | xz -T 0 > 
>> ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
>
> Since -T 0 will use all available CPUs (ie any attempts the user may
> have made to limit parallelism via BB_NUMBER_THREADS or PARALLEL_MAKE
> will be ignored), perhaps it's worth adding something like
> "--memlimit=70%" to try to give some protection for environments with
> lots of CPUs but not so much DRAM?

There's a few places where -T0 is passed already: tar.xz image
creation and opkg creation, so I guess we need to centralise this
somewhere too.

Ross
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] populate_sdk_base: use xz -T instead of pixz

[Andre McCurdy]

On Mon, Apr 9, 2018 at 8:19 AM, Ross Burton  wrote:
> xz has native support for threaded compression now and SDK creation was the 
> only
> part of oe-core which is using pixz instead of xz.
>
> Not only does this remove pixz-native from the SDK dependencies, but in my
> limited testing xz -T0 is slightly faster and produces smaller archives than
> pixz for the same input.
>
> Signed-off-by: Ross Burton 
> ---
>
>  # We want the MULTIARCH_TARGET_SYS to point to the TUNE_PKGARCH, not 
> PACKAGE_ARCH as it
> @@ -225,7 +225,7 @@ fakeroot tar_sdk() {
> # Package it up
> mkdir -p ${SDKDEPLOYDIR}
> cd ${SDK_OUTPUT}/${SDKPATH}
> -   tar ${SDKTAROPTS} -cf - . | pixz > 
> ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
> +   tar ${SDKTAROPTS} -cf - . | xz -T 0 > 
> ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz

Since -T 0 will use all available CPUs (ie any attempts the user may
have made to limit parallelism via BB_NUMBER_THREADS or PARALLEL_MAKE
will be ignored), perhaps it's worth adding something like
"--memlimit=70%" to try to give some protection for environments with
lots of CPUs but not so much DRAM?
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] populate_sdk_base: use xz -T instead of pixz

[Burton, Ross]

xz-native isn't in ASSUME_PROVIDED, we built it.

Ross

On 9 April 2018 at 19:44, Khem Raj  wrote:
> On Mon, Apr 9, 2018 at 11:29 AM, Burton, Ross  wrote:
>> No, it's SDK generation, so xz-native (see SDK_DEPENDS in the first hunk).
>>
>
> OK then we need to make sure that all  supported distros will have the
> threaded xz version
>
>> Ross
>>
>> On 9 April 2018 at 18:46, Khem Raj  wrote:
>>> On Mon, Apr 9, 2018 at 8:19 AM, Ross Burton  wrote:
 xz has native support for threaded compression now and SDK creation was 
 the only
 part of oe-core which is using pixz instead of xz.

 Not only does this remove pixz-native from the SDK dependencies, but in my
 limited testing xz -T0 is slightly faster and produces smaller archives 
 than
 pixz for the same input.

>>>
>>> this is using nativesdk-xz ?
>>>
 Signed-off-by: Ross Burton 
 ---
  meta/classes/populate_sdk_base.bbclass | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)

 diff --git a/meta/classes/populate_sdk_base.bbclass 
 b/meta/classes/populate_sdk_base.bbclass
 index 77ec8aaec27..79984d7914f 100644
 --- a/meta/classes/populate_sdk_base.bbclass
 +++ b/meta/classes/populate_sdk_base.bbclass
 @@ -46,7 +46,7 @@ TOOLCHAIN_TARGET_TASK_ATTEMPTONLY ?= ""
  TOOLCHAIN_OUTPUTNAME ?= "${SDK_NAME}-toolchain-${SDK_VERSION}"

  SDK_RDEPENDS = "${TOOLCHAIN_TARGET_TASK} ${TOOLCHAIN_HOST_TASK}"
 -SDK_DEPENDS = "virtual/fakeroot-native pixz-native cross-localedef-native 
 ${MLPREFIX}qemuwrapper-cross"
 +SDK_DEPENDS = "virtual/fakeroot-native xz-native cross-localedef-native 
 ${MLPREFIX}qemuwrapper-cross"
  SDK_DEPENDS_append_libc-glibc = " nativesdk-glibc-locale"

  # We want the MULTIARCH_TARGET_SYS to point to the TUNE_PKGARCH, not 
 PACKAGE_ARCH as it
 @@ -225,7 +225,7 @@ fakeroot tar_sdk() {
 # Package it up
 mkdir -p ${SDKDEPLOYDIR}
 cd ${SDK_OUTPUT}/${SDKPATH}
 -   tar ${SDKTAROPTS} -cf - . | pixz > 
 ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
 +   tar ${SDKTAROPTS} -cf - . | xz -T 0 > 
 ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
  }

  fakeroot create_shar() {
 --
 2.11.0

 --
 ___
 Openembedded-core mailing list
 Openembedded-core@lists.openembedded.org
 http://lists.openembedded.org/mailman/listinfo/openembedded-core
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] populate_sdk_base: use xz -T instead of pixz

[Khem Raj]

On Mon, Apr 9, 2018 at 11:29 AM, Burton, Ross  wrote:
> No, it's SDK generation, so xz-native (see SDK_DEPENDS in the first hunk).
>

OK then we need to make sure that all  supported distros will have the
threaded xz version

> Ross
>
> On 9 April 2018 at 18:46, Khem Raj  wrote:
>> On Mon, Apr 9, 2018 at 8:19 AM, Ross Burton  wrote:
>>> xz has native support for threaded compression now and SDK creation was the 
>>> only
>>> part of oe-core which is using pixz instead of xz.
>>>
>>> Not only does this remove pixz-native from the SDK dependencies, but in my
>>> limited testing xz -T0 is slightly faster and produces smaller archives than
>>> pixz for the same input.
>>>
>>
>> this is using nativesdk-xz ?
>>
>>> Signed-off-by: Ross Burton 
>>> ---
>>>  meta/classes/populate_sdk_base.bbclass | 4 ++--
>>>  1 file changed, 2 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/meta/classes/populate_sdk_base.bbclass 
>>> b/meta/classes/populate_sdk_base.bbclass
>>> index 77ec8aaec27..79984d7914f 100644
>>> --- a/meta/classes/populate_sdk_base.bbclass
>>> +++ b/meta/classes/populate_sdk_base.bbclass
>>> @@ -46,7 +46,7 @@ TOOLCHAIN_TARGET_TASK_ATTEMPTONLY ?= ""
>>>  TOOLCHAIN_OUTPUTNAME ?= "${SDK_NAME}-toolchain-${SDK_VERSION}"
>>>
>>>  SDK_RDEPENDS = "${TOOLCHAIN_TARGET_TASK} ${TOOLCHAIN_HOST_TASK}"
>>> -SDK_DEPENDS = "virtual/fakeroot-native pixz-native cross-localedef-native 
>>> ${MLPREFIX}qemuwrapper-cross"
>>> +SDK_DEPENDS = "virtual/fakeroot-native xz-native cross-localedef-native 
>>> ${MLPREFIX}qemuwrapper-cross"
>>>  SDK_DEPENDS_append_libc-glibc = " nativesdk-glibc-locale"
>>>
>>>  # We want the MULTIARCH_TARGET_SYS to point to the TUNE_PKGARCH, not 
>>> PACKAGE_ARCH as it
>>> @@ -225,7 +225,7 @@ fakeroot tar_sdk() {
>>> # Package it up
>>> mkdir -p ${SDKDEPLOYDIR}
>>> cd ${SDK_OUTPUT}/${SDKPATH}
>>> -   tar ${SDKTAROPTS} -cf - . | pixz > 
>>> ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
>>> +   tar ${SDKTAROPTS} -cf - . | xz -T 0 > 
>>> ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
>>>  }
>>>
>>>  fakeroot create_shar() {
>>> --
>>> 2.11.0
>>>
>>> --
>>> ___
>>> Openembedded-core mailing list
>>> Openembedded-core@lists.openembedded.org
>>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] populate_sdk_base: use xz -T instead of pixz

[Burton, Ross]

No, it's SDK generation, so xz-native (see SDK_DEPENDS in the first hunk).

Ross

On 9 April 2018 at 18:46, Khem Raj  wrote:
> On Mon, Apr 9, 2018 at 8:19 AM, Ross Burton  wrote:
>> xz has native support for threaded compression now and SDK creation was the 
>> only
>> part of oe-core which is using pixz instead of xz.
>>
>> Not only does this remove pixz-native from the SDK dependencies, but in my
>> limited testing xz -T0 is slightly faster and produces smaller archives than
>> pixz for the same input.
>>
>
> this is using nativesdk-xz ?
>
>> Signed-off-by: Ross Burton 
>> ---
>>  meta/classes/populate_sdk_base.bbclass | 4 ++--
>>  1 file changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/meta/classes/populate_sdk_base.bbclass 
>> b/meta/classes/populate_sdk_base.bbclass
>> index 77ec8aaec27..79984d7914f 100644
>> --- a/meta/classes/populate_sdk_base.bbclass
>> +++ b/meta/classes/populate_sdk_base.bbclass
>> @@ -46,7 +46,7 @@ TOOLCHAIN_TARGET_TASK_ATTEMPTONLY ?= ""
>>  TOOLCHAIN_OUTPUTNAME ?= "${SDK_NAME}-toolchain-${SDK_VERSION}"
>>
>>  SDK_RDEPENDS = "${TOOLCHAIN_TARGET_TASK} ${TOOLCHAIN_HOST_TASK}"
>> -SDK_DEPENDS = "virtual/fakeroot-native pixz-native cross-localedef-native 
>> ${MLPREFIX}qemuwrapper-cross"
>> +SDK_DEPENDS = "virtual/fakeroot-native xz-native cross-localedef-native 
>> ${MLPREFIX}qemuwrapper-cross"
>>  SDK_DEPENDS_append_libc-glibc = " nativesdk-glibc-locale"
>>
>>  # We want the MULTIARCH_TARGET_SYS to point to the TUNE_PKGARCH, not 
>> PACKAGE_ARCH as it
>> @@ -225,7 +225,7 @@ fakeroot tar_sdk() {
>> # Package it up
>> mkdir -p ${SDKDEPLOYDIR}
>> cd ${SDK_OUTPUT}/${SDKPATH}
>> -   tar ${SDKTAROPTS} -cf - . | pixz > 
>> ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
>> +   tar ${SDKTAROPTS} -cf - . | xz -T 0 > 
>> ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
>>  }
>>
>>  fakeroot create_shar() {
>> --
>> 2.11.0
>>
>> --
>> ___
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] populate_sdk_base: use xz -T instead of pixz

[Khem Raj]

On Mon, Apr 9, 2018 at 8:19 AM, Ross Burton  wrote:
> xz has native support for threaded compression now and SDK creation was the 
> only
> part of oe-core which is using pixz instead of xz.
>
> Not only does this remove pixz-native from the SDK dependencies, but in my
> limited testing xz -T0 is slightly faster and produces smaller archives than
> pixz for the same input.
>

this is using nativesdk-xz ?

> Signed-off-by: Ross Burton 
> ---
>  meta/classes/populate_sdk_base.bbclass | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/meta/classes/populate_sdk_base.bbclass 
> b/meta/classes/populate_sdk_base.bbclass
> index 77ec8aaec27..79984d7914f 100644
> --- a/meta/classes/populate_sdk_base.bbclass
> +++ b/meta/classes/populate_sdk_base.bbclass
> @@ -46,7 +46,7 @@ TOOLCHAIN_TARGET_TASK_ATTEMPTONLY ?= ""
>  TOOLCHAIN_OUTPUTNAME ?= "${SDK_NAME}-toolchain-${SDK_VERSION}"
>
>  SDK_RDEPENDS = "${TOOLCHAIN_TARGET_TASK} ${TOOLCHAIN_HOST_TASK}"
> -SDK_DEPENDS = "virtual/fakeroot-native pixz-native cross-localedef-native 
> ${MLPREFIX}qemuwrapper-cross"
> +SDK_DEPENDS = "virtual/fakeroot-native xz-native cross-localedef-native 
> ${MLPREFIX}qemuwrapper-cross"
>  SDK_DEPENDS_append_libc-glibc = " nativesdk-glibc-locale"
>
>  # We want the MULTIARCH_TARGET_SYS to point to the TUNE_PKGARCH, not 
> PACKAGE_ARCH as it
> @@ -225,7 +225,7 @@ fakeroot tar_sdk() {
> # Package it up
> mkdir -p ${SDKDEPLOYDIR}
> cd ${SDK_OUTPUT}/${SDKPATH}
> -   tar ${SDKTAROPTS} -cf - . | pixz > 
> ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
> +   tar ${SDKTAROPTS} -cf - . | xz -T 0 > 
> ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
>  }
>
>  fakeroot create_shar() {
> --
> 2.11.0
>
> --
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] populate_sdk_base: use xz -T instead of pixz

[Ross Burton]

xz has native support for threaded compression now and SDK creation was the only
part of oe-core which is using pixz instead of xz.

Not only does this remove pixz-native from the SDK dependencies, but in my
limited testing xz -T0 is slightly faster and produces smaller archives than
pixz for the same input.

Signed-off-by: Ross Burton 
---
 meta/classes/populate_sdk_base.bbclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/populate_sdk_base.bbclass 
b/meta/classes/populate_sdk_base.bbclass
index 77ec8aaec27..79984d7914f 100644
--- a/meta/classes/populate_sdk_base.bbclass
+++ b/meta/classes/populate_sdk_base.bbclass
@@ -46,7 +46,7 @@ TOOLCHAIN_TARGET_TASK_ATTEMPTONLY ?= ""
 TOOLCHAIN_OUTPUTNAME ?= "${SDK_NAME}-toolchain-${SDK_VERSION}"
 
 SDK_RDEPENDS = "${TOOLCHAIN_TARGET_TASK} ${TOOLCHAIN_HOST_TASK}"
-SDK_DEPENDS = "virtual/fakeroot-native pixz-native cross-localedef-native 
${MLPREFIX}qemuwrapper-cross"
+SDK_DEPENDS = "virtual/fakeroot-native xz-native cross-localedef-native 
${MLPREFIX}qemuwrapper-cross"
 SDK_DEPENDS_append_libc-glibc = " nativesdk-glibc-locale"
 
 # We want the MULTIARCH_TARGET_SYS to point to the TUNE_PKGARCH, not 
PACKAGE_ARCH as it
@@ -225,7 +225,7 @@ fakeroot tar_sdk() {
# Package it up
mkdir -p ${SDKDEPLOYDIR}
cd ${SDK_OUTPUT}/${SDKPATH}
-   tar ${SDKTAROPTS} -cf - . | pixz > 
${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
+   tar ${SDKTAROPTS} -cf - . | xz -T 0 > 
${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
 }
 
 fakeroot create_shar() {
-- 
2.11.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core