Re: [OE-core] [PATCH] populate_sdk_base: use xz -T instead of pixz
On 9 April 2018 at 22:20, Khem Raj wrote: > On Mon, Apr 9, 2018 at 1:44 PM, Burton, Ross wrote: >> On 9 April 2018 at 21:13, Andre McCurdy wrote: >>> On Mon, Apr 9, 2018 at 8:19 AM, Ross Burton wrote: xz has native support for threaded compression now and SDK creation was the only part of oe-core which is using pixz instead of xz. Not only does this remove pixz-native from the SDK dependencies, but in my limited testing xz -T0 is slightly faster and produces smaller archives than pixz for the same input. Signed-off-by: Ross Burton --- # We want the MULTIARCH_TARGET_SYS to point to the TUNE_PKGARCH, not PACKAGE_ARCH as it @@ -225,7 +225,7 @@ fakeroot tar_sdk() { # Package it up mkdir -p ${SDKDEPLOYDIR} cd ${SDK_OUTPUT}/${SDKPATH} - tar ${SDKTAROPTS} -cf - . | pixz > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz + tar ${SDKTAROPTS} -cf - . | xz -T 0 > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz >>> >>> Since -T 0 will use all available CPUs (ie any attempts the user may >>> have made to limit parallelism via BB_NUMBER_THREADS or PARALLEL_MAKE >>> will be ignored), perhaps it's worth adding something like >>> "--memlimit=70%" to try to give some protection for environments with >>> lots of CPUs but not so much DRAM? >> >> There's a few places where -T0 is passed already: tar.xz image >> creation and opkg creation, so I guess we need to centralise this >> somewhere too. >> > > May be use XZ_OPT env bariable and param to -T could use the bitbake > calculated number of CPUs -T0 is use as many threads as there are cores, which is the same as the bitbake function. Re-using BB_NUMBER_THREADS might work. Ross -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] populate_sdk_base: use xz -T instead of pixz
On Mon, Apr 9, 2018 at 1:44 PM, Burton, Ross wrote: > On 9 April 2018 at 21:13, Andre McCurdy wrote: >> On Mon, Apr 9, 2018 at 8:19 AM, Ross Burton wrote: >>> xz has native support for threaded compression now and SDK creation was the >>> only >>> part of oe-core which is using pixz instead of xz. >>> >>> Not only does this remove pixz-native from the SDK dependencies, but in my >>> limited testing xz -T0 is slightly faster and produces smaller archives than >>> pixz for the same input. >>> >>> Signed-off-by: Ross Burton >>> --- >>> >>> # We want the MULTIARCH_TARGET_SYS to point to the TUNE_PKGARCH, not >>> PACKAGE_ARCH as it >>> @@ -225,7 +225,7 @@ fakeroot tar_sdk() { >>> # Package it up >>> mkdir -p ${SDKDEPLOYDIR} >>> cd ${SDK_OUTPUT}/${SDKPATH} >>> - tar ${SDKTAROPTS} -cf - . | pixz > >>> ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz >>> + tar ${SDKTAROPTS} -cf - . | xz -T 0 > >>> ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz >> >> Since -T 0 will use all available CPUs (ie any attempts the user may >> have made to limit parallelism via BB_NUMBER_THREADS or PARALLEL_MAKE >> will be ignored), perhaps it's worth adding something like >> "--memlimit=70%" to try to give some protection for environments with >> lots of CPUs but not so much DRAM? > > There's a few places where -T0 is passed already: tar.xz image > creation and opkg creation, so I guess we need to centralise this > somewhere too. > May be use XZ_OPT env bariable and param to -T could use the bitbake calculated number of CPUs > Ross > -- > ___ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-core -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] populate_sdk_base: use xz -T instead of pixz
On 9 April 2018 at 21:13, Andre McCurdy wrote: > On Mon, Apr 9, 2018 at 8:19 AM, Ross Burton wrote: >> xz has native support for threaded compression now and SDK creation was the >> only >> part of oe-core which is using pixz instead of xz. >> >> Not only does this remove pixz-native from the SDK dependencies, but in my >> limited testing xz -T0 is slightly faster and produces smaller archives than >> pixz for the same input. >> >> Signed-off-by: Ross Burton >> --- >> >> # We want the MULTIARCH_TARGET_SYS to point to the TUNE_PKGARCH, not >> PACKAGE_ARCH as it >> @@ -225,7 +225,7 @@ fakeroot tar_sdk() { >> # Package it up >> mkdir -p ${SDKDEPLOYDIR} >> cd ${SDK_OUTPUT}/${SDKPATH} >> - tar ${SDKTAROPTS} -cf - . | pixz > >> ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz >> + tar ${SDKTAROPTS} -cf - . | xz -T 0 > >> ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz > > Since -T 0 will use all available CPUs (ie any attempts the user may > have made to limit parallelism via BB_NUMBER_THREADS or PARALLEL_MAKE > will be ignored), perhaps it's worth adding something like > "--memlimit=70%" to try to give some protection for environments with > lots of CPUs but not so much DRAM? There's a few places where -T0 is passed already: tar.xz image creation and opkg creation, so I guess we need to centralise this somewhere too. Ross -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] populate_sdk_base: use xz -T instead of pixz
On Mon, Apr 9, 2018 at 8:19 AM, Ross Burton wrote: > xz has native support for threaded compression now and SDK creation was the > only > part of oe-core which is using pixz instead of xz. > > Not only does this remove pixz-native from the SDK dependencies, but in my > limited testing xz -T0 is slightly faster and produces smaller archives than > pixz for the same input. > > Signed-off-by: Ross Burton > --- > > # We want the MULTIARCH_TARGET_SYS to point to the TUNE_PKGARCH, not > PACKAGE_ARCH as it > @@ -225,7 +225,7 @@ fakeroot tar_sdk() { > # Package it up > mkdir -p ${SDKDEPLOYDIR} > cd ${SDK_OUTPUT}/${SDKPATH} > - tar ${SDKTAROPTS} -cf - . | pixz > > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz > + tar ${SDKTAROPTS} -cf - . | xz -T 0 > > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz Since -T 0 will use all available CPUs (ie any attempts the user may have made to limit parallelism via BB_NUMBER_THREADS or PARALLEL_MAKE will be ignored), perhaps it's worth adding something like "--memlimit=70%" to try to give some protection for environments with lots of CPUs but not so much DRAM? -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] populate_sdk_base: use xz -T instead of pixz
xz-native isn't in ASSUME_PROVIDED, we built it. Ross On 9 April 2018 at 19:44, Khem Raj wrote: > On Mon, Apr 9, 2018 at 11:29 AM, Burton, Ross wrote: >> No, it's SDK generation, so xz-native (see SDK_DEPENDS in the first hunk). >> > > OK then we need to make sure that all supported distros will have the > threaded xz version > >> Ross >> >> On 9 April 2018 at 18:46, Khem Raj wrote: >>> On Mon, Apr 9, 2018 at 8:19 AM, Ross Burton wrote: xz has native support for threaded compression now and SDK creation was the only part of oe-core which is using pixz instead of xz. Not only does this remove pixz-native from the SDK dependencies, but in my limited testing xz -T0 is slightly faster and produces smaller archives than pixz for the same input. >>> >>> this is using nativesdk-xz ? >>> Signed-off-by: Ross Burton --- meta/classes/populate_sdk_base.bbclass | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/meta/classes/populate_sdk_base.bbclass b/meta/classes/populate_sdk_base.bbclass index 77ec8aaec27..79984d7914f 100644 --- a/meta/classes/populate_sdk_base.bbclass +++ b/meta/classes/populate_sdk_base.bbclass @@ -46,7 +46,7 @@ TOOLCHAIN_TARGET_TASK_ATTEMPTONLY ?= "" TOOLCHAIN_OUTPUTNAME ?= "${SDK_NAME}-toolchain-${SDK_VERSION}" SDK_RDEPENDS = "${TOOLCHAIN_TARGET_TASK} ${TOOLCHAIN_HOST_TASK}" -SDK_DEPENDS = "virtual/fakeroot-native pixz-native cross-localedef-native ${MLPREFIX}qemuwrapper-cross" +SDK_DEPENDS = "virtual/fakeroot-native xz-native cross-localedef-native ${MLPREFIX}qemuwrapper-cross" SDK_DEPENDS_append_libc-glibc = " nativesdk-glibc-locale" # We want the MULTIARCH_TARGET_SYS to point to the TUNE_PKGARCH, not PACKAGE_ARCH as it @@ -225,7 +225,7 @@ fakeroot tar_sdk() { # Package it up mkdir -p ${SDKDEPLOYDIR} cd ${SDK_OUTPUT}/${SDKPATH} - tar ${SDKTAROPTS} -cf - . | pixz > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz + tar ${SDKTAROPTS} -cf - . | xz -T 0 > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz } fakeroot create_shar() { -- 2.11.0 -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] populate_sdk_base: use xz -T instead of pixz
On Mon, Apr 9, 2018 at 11:29 AM, Burton, Ross wrote: > No, it's SDK generation, so xz-native (see SDK_DEPENDS in the first hunk). > OK then we need to make sure that all supported distros will have the threaded xz version > Ross > > On 9 April 2018 at 18:46, Khem Raj wrote: >> On Mon, Apr 9, 2018 at 8:19 AM, Ross Burton wrote: >>> xz has native support for threaded compression now and SDK creation was the >>> only >>> part of oe-core which is using pixz instead of xz. >>> >>> Not only does this remove pixz-native from the SDK dependencies, but in my >>> limited testing xz -T0 is slightly faster and produces smaller archives than >>> pixz for the same input. >>> >> >> this is using nativesdk-xz ? >> >>> Signed-off-by: Ross Burton >>> --- >>> meta/classes/populate_sdk_base.bbclass | 4 ++-- >>> 1 file changed, 2 insertions(+), 2 deletions(-) >>> >>> diff --git a/meta/classes/populate_sdk_base.bbclass >>> b/meta/classes/populate_sdk_base.bbclass >>> index 77ec8aaec27..79984d7914f 100644 >>> --- a/meta/classes/populate_sdk_base.bbclass >>> +++ b/meta/classes/populate_sdk_base.bbclass >>> @@ -46,7 +46,7 @@ TOOLCHAIN_TARGET_TASK_ATTEMPTONLY ?= "" >>> TOOLCHAIN_OUTPUTNAME ?= "${SDK_NAME}-toolchain-${SDK_VERSION}" >>> >>> SDK_RDEPENDS = "${TOOLCHAIN_TARGET_TASK} ${TOOLCHAIN_HOST_TASK}" >>> -SDK_DEPENDS = "virtual/fakeroot-native pixz-native cross-localedef-native >>> ${MLPREFIX}qemuwrapper-cross" >>> +SDK_DEPENDS = "virtual/fakeroot-native xz-native cross-localedef-native >>> ${MLPREFIX}qemuwrapper-cross" >>> SDK_DEPENDS_append_libc-glibc = " nativesdk-glibc-locale" >>> >>> # We want the MULTIARCH_TARGET_SYS to point to the TUNE_PKGARCH, not >>> PACKAGE_ARCH as it >>> @@ -225,7 +225,7 @@ fakeroot tar_sdk() { >>> # Package it up >>> mkdir -p ${SDKDEPLOYDIR} >>> cd ${SDK_OUTPUT}/${SDKPATH} >>> - tar ${SDKTAROPTS} -cf - . | pixz > >>> ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz >>> + tar ${SDKTAROPTS} -cf - . | xz -T 0 > >>> ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz >>> } >>> >>> fakeroot create_shar() { >>> -- >>> 2.11.0 >>> >>> -- >>> ___ >>> Openembedded-core mailing list >>> Openembedded-core@lists.openembedded.org >>> http://lists.openembedded.org/mailman/listinfo/openembedded-core -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] populate_sdk_base: use xz -T instead of pixz
No, it's SDK generation, so xz-native (see SDK_DEPENDS in the first hunk). Ross On 9 April 2018 at 18:46, Khem Raj wrote: > On Mon, Apr 9, 2018 at 8:19 AM, Ross Burton wrote: >> xz has native support for threaded compression now and SDK creation was the >> only >> part of oe-core which is using pixz instead of xz. >> >> Not only does this remove pixz-native from the SDK dependencies, but in my >> limited testing xz -T0 is slightly faster and produces smaller archives than >> pixz for the same input. >> > > this is using nativesdk-xz ? > >> Signed-off-by: Ross Burton >> --- >> meta/classes/populate_sdk_base.bbclass | 4 ++-- >> 1 file changed, 2 insertions(+), 2 deletions(-) >> >> diff --git a/meta/classes/populate_sdk_base.bbclass >> b/meta/classes/populate_sdk_base.bbclass >> index 77ec8aaec27..79984d7914f 100644 >> --- a/meta/classes/populate_sdk_base.bbclass >> +++ b/meta/classes/populate_sdk_base.bbclass >> @@ -46,7 +46,7 @@ TOOLCHAIN_TARGET_TASK_ATTEMPTONLY ?= "" >> TOOLCHAIN_OUTPUTNAME ?= "${SDK_NAME}-toolchain-${SDK_VERSION}" >> >> SDK_RDEPENDS = "${TOOLCHAIN_TARGET_TASK} ${TOOLCHAIN_HOST_TASK}" >> -SDK_DEPENDS = "virtual/fakeroot-native pixz-native cross-localedef-native >> ${MLPREFIX}qemuwrapper-cross" >> +SDK_DEPENDS = "virtual/fakeroot-native xz-native cross-localedef-native >> ${MLPREFIX}qemuwrapper-cross" >> SDK_DEPENDS_append_libc-glibc = " nativesdk-glibc-locale" >> >> # We want the MULTIARCH_TARGET_SYS to point to the TUNE_PKGARCH, not >> PACKAGE_ARCH as it >> @@ -225,7 +225,7 @@ fakeroot tar_sdk() { >> # Package it up >> mkdir -p ${SDKDEPLOYDIR} >> cd ${SDK_OUTPUT}/${SDKPATH} >> - tar ${SDKTAROPTS} -cf - . | pixz > >> ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz >> + tar ${SDKTAROPTS} -cf - . | xz -T 0 > >> ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz >> } >> >> fakeroot create_shar() { >> -- >> 2.11.0 >> >> -- >> ___ >> Openembedded-core mailing list >> Openembedded-core@lists.openembedded.org >> http://lists.openembedded.org/mailman/listinfo/openembedded-core -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] populate_sdk_base: use xz -T instead of pixz
On Mon, Apr 9, 2018 at 8:19 AM, Ross Burton wrote: > xz has native support for threaded compression now and SDK creation was the > only > part of oe-core which is using pixz instead of xz. > > Not only does this remove pixz-native from the SDK dependencies, but in my > limited testing xz -T0 is slightly faster and produces smaller archives than > pixz for the same input. > this is using nativesdk-xz ? > Signed-off-by: Ross Burton > --- > meta/classes/populate_sdk_base.bbclass | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/meta/classes/populate_sdk_base.bbclass > b/meta/classes/populate_sdk_base.bbclass > index 77ec8aaec27..79984d7914f 100644 > --- a/meta/classes/populate_sdk_base.bbclass > +++ b/meta/classes/populate_sdk_base.bbclass > @@ -46,7 +46,7 @@ TOOLCHAIN_TARGET_TASK_ATTEMPTONLY ?= "" > TOOLCHAIN_OUTPUTNAME ?= "${SDK_NAME}-toolchain-${SDK_VERSION}" > > SDK_RDEPENDS = "${TOOLCHAIN_TARGET_TASK} ${TOOLCHAIN_HOST_TASK}" > -SDK_DEPENDS = "virtual/fakeroot-native pixz-native cross-localedef-native > ${MLPREFIX}qemuwrapper-cross" > +SDK_DEPENDS = "virtual/fakeroot-native xz-native cross-localedef-native > ${MLPREFIX}qemuwrapper-cross" > SDK_DEPENDS_append_libc-glibc = " nativesdk-glibc-locale" > > # We want the MULTIARCH_TARGET_SYS to point to the TUNE_PKGARCH, not > PACKAGE_ARCH as it > @@ -225,7 +225,7 @@ fakeroot tar_sdk() { > # Package it up > mkdir -p ${SDKDEPLOYDIR} > cd ${SDK_OUTPUT}/${SDKPATH} > - tar ${SDKTAROPTS} -cf - . | pixz > > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz > + tar ${SDKTAROPTS} -cf - . | xz -T 0 > > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz > } > > fakeroot create_shar() { > -- > 2.11.0 > > -- > ___ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-core -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core