Re: [OE-core] [PATCH] image_types: use pigz to create .gz files

2018-04-13 Thread Richard Purdie 
On Wed, 2018-04-11 at 14:36 +0100, Burton, Ross wrote:
> Wouldn't that introduce a fun circular dependency if pigz-native
> needs
> to be built?

For the sstate code we could probably make it use pigz if present. 

We could also only depend on pigz-native for target, or encode the
actual pigz-native dependencies.

So there may be a few options if its worth the performance win.

Cheers,

Richard
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] image_types: use pigz to create .gz files

2018-04-11 Thread Khem Raj 
On Wed, Apr 11, 2018 at 6:19 PM, Denys Dmytriyenko  wrote:
> On Wed, Apr 11, 2018 at 05:07:38PM +0100, Burton, Ross wrote:
>> Woud be interesting to see a benchmark of a full build from scratch
>> with host gzip vs host pigz...
>
> Is pigz so widely used now to be available in every Linux distro like gzip?
>

Tested distros do seem to provide it as old as ubuntu 14.04 and centos 6
xz is another option for sstate archives the extra time it requires might
be compensated with threaded run in addition the archives will be smaller

>
>> On 11 April 2018 at 17:03, Khem Raj  wrote:
>> > On Wed, Apr 11, 2018 at 6:36 AM, Burton, Ross  
>> > wrote:
>> >> Wouldn't that introduce a fun circular dependency if pigz-native needs
>> >> to be built?
>> >>
>> >
>> > interesting idea certainly, if it ends up with improvements in build
>> > time, we could
>> > add pigz to host dependencies.
>> >
>> >> Ross
>> >>
>> >> On 11 April 2018 at 11:53, Martin Jansa  wrote:
>> >>> What about using pigz for sstate-cache archives?
>> >>>
>> >>> On Tue, Apr 10, 2018 at 4:35 PM, Ross Burton  
>> >>> wrote:
>> 
>>  Since pigz is no longer a drop-in replacement for gzip (oe-core 1624b7b)
>>  the
>>  image creation has been using gzip instead of pigz, despite still
>>  depending on
>>  pigz-native.  Fix this by invoking pigz explicitly.
>> 
>>  Signed-off-by: Ross Burton 
>>  ---
>>   meta/classes/image_types.bbclass | 2 +-
>>   1 file changed, 1 insertion(+), 1 deletion(-)
>> 
>>  diff --git a/meta/classes/image_types.bbclass
>>  b/meta/classes/image_types.bbclass
>>  index e872ae20982..00a00d318f4 100644
>>  --- a/meta/classes/image_types.bbclass
>>  +++ b/meta/classes/image_types.bbclass
>>  @@ -281,7 +281,7 @@ COMPRESSIONTYPES ?= ""
>> 
>>   CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum
>>  sha224sum sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2
>>  ${COMPRESSIONTYPES}"
>>   CONVERSION_CMD_lzma = "lzma -k -f -7
>>  ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
>>  -CONVERSION_CMD_gz = "gzip -f -9 -n -c
>>  ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} >
>>  ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
>>  +CONVERSION_CMD_gz = "pigz -f -9 -n -c
>>  ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} >
>>  ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
>>   CONVERSION_CMD_bz2 = "pbzip2 -f -k
>>  ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
>>   CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_THREADS}
>>  --check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} 
>>  >
>>  ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.xz"
>>   CONVERSION_CMD_lz4 = "lz4 -9 -z -l
>>  ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}
>>  ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"
>>  --
>>  2.11.0
>> 
>>  --
>>  ___
>>  Openembedded-core mailing list
>>  Openembedded-core@lists.openembedded.org
>>  http://lists.openembedded.org/mailman/listinfo/openembedded-core
>> >>>
>> >>>
>> >> --
>> >> ___
>> >> Openembedded-core mailing list
>> >> Openembedded-core@lists.openembedded.org
>> >> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>> --
>> ___
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] image_types: use pigz to create .gz files

2018-04-11 Thread Denys Dmytriyenko 
On Wed, Apr 11, 2018 at 05:07:38PM +0100, Burton, Ross wrote:
> Woud be interesting to see a benchmark of a full build from scratch
> with host gzip vs host pigz...

Is pigz so widely used now to be available in every Linux distro like gzip?


> On 11 April 2018 at 17:03, Khem Raj  wrote:
> > On Wed, Apr 11, 2018 at 6:36 AM, Burton, Ross  wrote:
> >> Wouldn't that introduce a fun circular dependency if pigz-native needs
> >> to be built?
> >>
> >
> > interesting idea certainly, if it ends up with improvements in build
> > time, we could
> > add pigz to host dependencies.
> >
> >> Ross
> >>
> >> On 11 April 2018 at 11:53, Martin Jansa  wrote:
> >>> What about using pigz for sstate-cache archives?
> >>>
> >>> On Tue, Apr 10, 2018 at 4:35 PM, Ross Burton  
> >>> wrote:
> 
>  Since pigz is no longer a drop-in replacement for gzip (oe-core 1624b7b)
>  the
>  image creation has been using gzip instead of pigz, despite still
>  depending on
>  pigz-native.  Fix this by invoking pigz explicitly.
> 
>  Signed-off-by: Ross Burton 
>  ---
>   meta/classes/image_types.bbclass | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
>  diff --git a/meta/classes/image_types.bbclass
>  b/meta/classes/image_types.bbclass
>  index e872ae20982..00a00d318f4 100644
>  --- a/meta/classes/image_types.bbclass
>  +++ b/meta/classes/image_types.bbclass
>  @@ -281,7 +281,7 @@ COMPRESSIONTYPES ?= ""
> 
>   CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum
>  sha224sum sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2
>  ${COMPRESSIONTYPES}"
>   CONVERSION_CMD_lzma = "lzma -k -f -7
>  ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
>  -CONVERSION_CMD_gz = "gzip -f -9 -n -c
>  ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} >
>  ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
>  +CONVERSION_CMD_gz = "pigz -f -9 -n -c
>  ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} >
>  ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
>   CONVERSION_CMD_bz2 = "pbzip2 -f -k
>  ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
>   CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_THREADS}
>  --check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} >
>  ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.xz"
>   CONVERSION_CMD_lz4 = "lz4 -9 -z -l
>  ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}
>  ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"
>  --
>  2.11.0
> 
>  --
>  ___
>  Openembedded-core mailing list
>  Openembedded-core@lists.openembedded.org
>  http://lists.openembedded.org/mailman/listinfo/openembedded-core
> >>>
> >>>
> >> --
> >> ___
> >> Openembedded-core mailing list
> >> Openembedded-core@lists.openembedded.org
> >> http://lists.openembedded.org/mailman/listinfo/openembedded-core
> -- 
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] image_types: use pigz to create .gz files

2018-04-11 Thread Burton, Ross 
Woud be interesting to see a benchmark of a full build from scratch
with host gzip vs host pigz...

On 11 April 2018 at 17:03, Khem Raj  wrote:
> On Wed, Apr 11, 2018 at 6:36 AM, Burton, Ross  wrote:
>> Wouldn't that introduce a fun circular dependency if pigz-native needs
>> to be built?
>>
>
> interesting idea certainly, if it ends up with improvements in build
> time, we could
> add pigz to host dependencies.
>
>> Ross
>>
>> On 11 April 2018 at 11:53, Martin Jansa  wrote:
>>> What about using pigz for sstate-cache archives?
>>>
>>> On Tue, Apr 10, 2018 at 4:35 PM, Ross Burton  wrote:

 Since pigz is no longer a drop-in replacement for gzip (oe-core 1624b7b)
 the
 image creation has been using gzip instead of pigz, despite still
 depending on
 pigz-native.  Fix this by invoking pigz explicitly.

 Signed-off-by: Ross Burton 
 ---
  meta/classes/image_types.bbclass | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

 diff --git a/meta/classes/image_types.bbclass
 b/meta/classes/image_types.bbclass
 index e872ae20982..00a00d318f4 100644
 --- a/meta/classes/image_types.bbclass
 +++ b/meta/classes/image_types.bbclass
 @@ -281,7 +281,7 @@ COMPRESSIONTYPES ?= ""

  CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum
 sha224sum sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2
 ${COMPRESSIONTYPES}"
  CONVERSION_CMD_lzma = "lzma -k -f -7
 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
 -CONVERSION_CMD_gz = "gzip -f -9 -n -c
 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} >
 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
 +CONVERSION_CMD_gz = "pigz -f -9 -n -c
 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} >
 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
  CONVERSION_CMD_bz2 = "pbzip2 -f -k
 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
  CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_THREADS}
 --check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} >
 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.xz"
  CONVERSION_CMD_lz4 = "lz4 -9 -z -l
 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}
 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"
 --
 2.11.0

 --
 ___
 Openembedded-core mailing list
 Openembedded-core@lists.openembedded.org
 http://lists.openembedded.org/mailman/listinfo/openembedded-core
>>>
>>>
>> --
>> ___
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] image_types: use pigz to create .gz files

2018-04-11 Thread Khem Raj 
On Wed, Apr 11, 2018 at 6:36 AM, Burton, Ross  wrote:
> Wouldn't that introduce a fun circular dependency if pigz-native needs
> to be built?
>

interesting idea certainly, if it ends up with improvements in build
time, we could
add pigz to host dependencies.

> Ross
>
> On 11 April 2018 at 11:53, Martin Jansa  wrote:
>> What about using pigz for sstate-cache archives?
>>
>> On Tue, Apr 10, 2018 at 4:35 PM, Ross Burton  wrote:
>>>
>>> Since pigz is no longer a drop-in replacement for gzip (oe-core 1624b7b)
>>> the
>>> image creation has been using gzip instead of pigz, despite still
>>> depending on
>>> pigz-native.  Fix this by invoking pigz explicitly.
>>>
>>> Signed-off-by: Ross Burton 
>>> ---
>>>  meta/classes/image_types.bbclass | 2 +-
>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/meta/classes/image_types.bbclass
>>> b/meta/classes/image_types.bbclass
>>> index e872ae20982..00a00d318f4 100644
>>> --- a/meta/classes/image_types.bbclass
>>> +++ b/meta/classes/image_types.bbclass
>>> @@ -281,7 +281,7 @@ COMPRESSIONTYPES ?= ""
>>>
>>>  CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum
>>> sha224sum sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2
>>> ${COMPRESSIONTYPES}"
>>>  CONVERSION_CMD_lzma = "lzma -k -f -7
>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
>>> -CONVERSION_CMD_gz = "gzip -f -9 -n -c
>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} >
>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
>>> +CONVERSION_CMD_gz = "pigz -f -9 -n -c
>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} >
>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
>>>  CONVERSION_CMD_bz2 = "pbzip2 -f -k
>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
>>>  CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_THREADS}
>>> --check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} >
>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.xz"
>>>  CONVERSION_CMD_lz4 = "lz4 -9 -z -l
>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}
>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"
>>> --
>>> 2.11.0
>>>
>>> --
>>> ___
>>> Openembedded-core mailing list
>>> Openembedded-core@lists.openembedded.org
>>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>>
>>
> --
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] image_types: use pigz to create .gz files

2018-04-11 Thread Burton, Ross 
Wouldn't that introduce a fun circular dependency if pigz-native needs
to be built?

Ross

On 11 April 2018 at 11:53, Martin Jansa  wrote:
> What about using pigz for sstate-cache archives?
>
> On Tue, Apr 10, 2018 at 4:35 PM, Ross Burton  wrote:
>>
>> Since pigz is no longer a drop-in replacement for gzip (oe-core 1624b7b)
>> the
>> image creation has been using gzip instead of pigz, despite still
>> depending on
>> pigz-native.  Fix this by invoking pigz explicitly.
>>
>> Signed-off-by: Ross Burton 
>> ---
>>  meta/classes/image_types.bbclass | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/meta/classes/image_types.bbclass
>> b/meta/classes/image_types.bbclass
>> index e872ae20982..00a00d318f4 100644
>> --- a/meta/classes/image_types.bbclass
>> +++ b/meta/classes/image_types.bbclass
>> @@ -281,7 +281,7 @@ COMPRESSIONTYPES ?= ""
>>
>>  CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum
>> sha224sum sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2
>> ${COMPRESSIONTYPES}"
>>  CONVERSION_CMD_lzma = "lzma -k -f -7
>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
>> -CONVERSION_CMD_gz = "gzip -f -9 -n -c
>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} >
>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
>> +CONVERSION_CMD_gz = "pigz -f -9 -n -c
>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} >
>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
>>  CONVERSION_CMD_bz2 = "pbzip2 -f -k
>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
>>  CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_THREADS}
>> --check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} >
>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.xz"
>>  CONVERSION_CMD_lz4 = "lz4 -9 -z -l
>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}
>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"
>> --
>> 2.11.0
>>
>> --
>> ___
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
>
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] image_types: use pigz to create .gz files

2018-04-11 Thread Martin Jansa 
What about using pigz for sstate-cache archives?

On Tue, Apr 10, 2018 at 4:35 PM, Ross Burton  wrote:

> Since pigz is no longer a drop-in replacement for gzip (oe-core 1624b7b)
> the
> image creation has been using gzip instead of pigz, despite still
> depending on
> pigz-native.  Fix this by invoking pigz explicitly.
>
> Signed-off-by: Ross Burton 
> ---
>  meta/classes/image_types.bbclass | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.
> bbclass
> index e872ae20982..00a00d318f4 100644
> --- a/meta/classes/image_types.bbclass
> +++ b/meta/classes/image_types.bbclass
> @@ -281,7 +281,7 @@ COMPRESSIONTYPES ?= ""
>
>  CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum
> sha224sum sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2
> ${COMPRESSIONTYPES}"
>  CONVERSION_CMD_lzma = "lzma -k -f -7 ${IMAGE_NAME}${IMAGE_NAME_
> SUFFIX}.${type}"
> -CONVERSION_CMD_gz = "gzip -f -9 -n -c 
> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}
> > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
> +CONVERSION_CMD_gz = "pigz -f -9 -n -c 
> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}
> > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
>  CONVERSION_CMD_bz2 = "pbzip2 -f -k ${IMAGE_NAME}${IMAGE_NAME_
> SUFFIX}.${type}"
>  CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_THREADS}
> --check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} >
> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.xz"
>  CONVERSION_CMD_lz4 = "lz4 -9 -z -l ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}
> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"
> --
> 2.11.0
>
> --
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core