Re: [OE-core] [PATCH] image_types: add Zstandard conversion support

2019-11-06 Thread Stefan Agner 
Hi Andre,

On 2019-11-06 02:25, Andre McCurdy wrote:
> On Tue, Nov 5, 2019 at 3:13 PM Stefan Agner  wrote:
>>
>> From: Stefan Agner 
>>
>> Add Zstandard (or just Zstd) compression support. This allows to
>> create Zstd compressed tarballs by using tar.zst as IMAGE_FSTYPES.
>>
>> Signed-off-by: Stefan Agner 
>> ---
>>  meta/classes/image_types.bbclass | 6 --
>>  1 file changed, 4 insertions(+), 2 deletions(-)
>>
>> diff --git a/meta/classes/image_types.bbclass 
>> b/meta/classes/image_types.bbclass
>> index 2eeffbb366..eeae652e02 100644
>> --- a/meta/classes/image_types.bbclass
>> +++ b/meta/classes/image_types.bbclass
>> @@ -269,7 +269,7 @@ IMAGE_TYPES = " \
>>  hddimg \
>>  squashfs squashfs-xz squashfs-lzo squashfs-lz4 \
>>  ubi ubifs multiubi \
>> -tar tar.gz tar.bz2 tar.xz tar.lz4 \
>> +tar tar.gz tar.bz2 tar.xz tar.lz4 tar.zst \
>>  cpio cpio.gz cpio.xz cpio.lzma cpio.lz4 \
>>  wic wic.gz wic.bz2 wic.lzma \
>>  container \
>> @@ -282,7 +282,7 @@ IMAGE_TYPES = " \
>>  # CONVERSION_CMD/DEPENDS.
>>  COMPRESSIONTYPES ?= ""
>>
>> -CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum sha224sum 
>> sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2 base64 
>> ${COMPRESSIONTYPES}"
>> +CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip zst sum md5sum sha1sum 
>> sha224sum sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2 base64 
>> ${COMPRESSIONTYPES}"
>>  CONVERSION_CMD_lzma = "lzma -k -f -7 
>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
>>  CONVERSION_CMD_gz = "gzip -f -9 -n -c --rsyncable 
>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > 
>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
>>  CONVERSION_CMD_bz2 = "pbzip2 -f -k 
>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
>> @@ -290,6 +290,7 @@ CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} 
>> ${XZ_DEFAULTS} --check=
>>  CONVERSION_CMD_lz4 = "lz4 -9 -z -l 
>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} 
>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"
>>  CONVERSION_CMD_lzo = "lzop -9 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
>>  CONVERSION_CMD_zip = "zip ${ZIP_COMPRESSION_LEVEL} 
>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.zip 
>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
>> +CONVERSION_CMD_zst = "zstd -f -k -c ${ZSTD_COMPRESSION_LEVEL} 
>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > 
>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.zst"
> 
> Where is ZSTD_COMPRESSION_LEVEL defined?
> 

Oh good catch, I got distracted halfway during the implementation it
seem :-)

I guess since it was not defined it did not break...

Will send a v2.

--
Stefan

>>  CONVERSION_CMD_sum = "sumtool -i ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} 
>> -o ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sum ${JFFS2_SUM_EXTRA_ARGS}"
>>  CONVERSION_CMD_md5sum = "md5sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > 
>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.md5sum"
>>  CONVERSION_CMD_sha1sum = "sha1sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} 
>> > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sha1sum"
>> @@ -310,6 +311,7 @@ CONVERSION_DEPENDS_xz = "xz-native"
>>  CONVERSION_DEPENDS_lz4 = "lz4-native"
>>  CONVERSION_DEPENDS_lzo = "lzop-native"
>>  CONVERSION_DEPENDS_zip = "zip-native"
>> +CONVERSION_DEPENDS_zst = "zstd-native"
>>  CONVERSION_DEPENDS_sum = "mtd-utils-native"
>>  CONVERSION_DEPENDS_bmap = "bmap-tools-native"
>>  CONVERSION_DEPENDS_u-boot = "u-boot-tools-native"
>> --
>> 2.17.1
>>
>> --
>> ___
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] image_types: add Zstandard conversion support

2019-11-05 Thread Andre McCurdy 
On Tue, Nov 5, 2019 at 3:13 PM Stefan Agner  wrote:
>
> From: Stefan Agner 
>
> Add Zstandard (or just Zstd) compression support. This allows to
> create Zstd compressed tarballs by using tar.zst as IMAGE_FSTYPES.
>
> Signed-off-by: Stefan Agner 
> ---
>  meta/classes/image_types.bbclass | 6 --
>  1 file changed, 4 insertions(+), 2 deletions(-)
>
> diff --git a/meta/classes/image_types.bbclass 
> b/meta/classes/image_types.bbclass
> index 2eeffbb366..eeae652e02 100644
> --- a/meta/classes/image_types.bbclass
> +++ b/meta/classes/image_types.bbclass
> @@ -269,7 +269,7 @@ IMAGE_TYPES = " \
>  hddimg \
>  squashfs squashfs-xz squashfs-lzo squashfs-lz4 \
>  ubi ubifs multiubi \
> -tar tar.gz tar.bz2 tar.xz tar.lz4 \
> +tar tar.gz tar.bz2 tar.xz tar.lz4 tar.zst \
>  cpio cpio.gz cpio.xz cpio.lzma cpio.lz4 \
>  wic wic.gz wic.bz2 wic.lzma \
>  container \
> @@ -282,7 +282,7 @@ IMAGE_TYPES = " \
>  # CONVERSION_CMD/DEPENDS.
>  COMPRESSIONTYPES ?= ""
>
> -CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum sha224sum 
> sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2 base64 
> ${COMPRESSIONTYPES}"
> +CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip zst sum md5sum sha1sum 
> sha224sum sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2 base64 
> ${COMPRESSIONTYPES}"
>  CONVERSION_CMD_lzma = "lzma -k -f -7 
> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
>  CONVERSION_CMD_gz = "gzip -f -9 -n -c --rsyncable 
> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > 
> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
>  CONVERSION_CMD_bz2 = "pbzip2 -f -k ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
> @@ -290,6 +290,7 @@ CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} 
> ${XZ_DEFAULTS} --check=
>  CONVERSION_CMD_lz4 = "lz4 -9 -z -l ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} 
> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"
>  CONVERSION_CMD_lzo = "lzop -9 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
>  CONVERSION_CMD_zip = "zip ${ZIP_COMPRESSION_LEVEL} 
> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.zip 
> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
> +CONVERSION_CMD_zst = "zstd -f -k -c ${ZSTD_COMPRESSION_LEVEL} 
> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > 
> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.zst"

Where is ZSTD_COMPRESSION_LEVEL defined?

>  CONVERSION_CMD_sum = "sumtool -i ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} 
> -o ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sum ${JFFS2_SUM_EXTRA_ARGS}"
>  CONVERSION_CMD_md5sum = "md5sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > 
> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.md5sum"
>  CONVERSION_CMD_sha1sum = "sha1sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} 
> > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sha1sum"
> @@ -310,6 +311,7 @@ CONVERSION_DEPENDS_xz = "xz-native"
>  CONVERSION_DEPENDS_lz4 = "lz4-native"
>  CONVERSION_DEPENDS_lzo = "lzop-native"
>  CONVERSION_DEPENDS_zip = "zip-native"
> +CONVERSION_DEPENDS_zst = "zstd-native"
>  CONVERSION_DEPENDS_sum = "mtd-utils-native"
>  CONVERSION_DEPENDS_bmap = "bmap-tools-native"
>  CONVERSION_DEPENDS_u-boot = "u-boot-tools-native"
> --
> 2.17.1
>
> --
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] image_types: add Zstandard conversion support

2019-11-05 Thread Khem Raj 
On Tue, Nov 5, 2019 at 3:13 PM Stefan Agner  wrote:
>
> From: Stefan Agner 
>
> Add Zstandard (or just Zstd) compression support. This allows to
> create Zstd compressed tarballs by using tar.zst as IMAGE_FSTYPES.
>

this would need documentation changes too I guess

> Signed-off-by: Stefan Agner 
> ---
>  meta/classes/image_types.bbclass | 6 --
>  1 file changed, 4 insertions(+), 2 deletions(-)
>
> diff --git a/meta/classes/image_types.bbclass 
> b/meta/classes/image_types.bbclass
> index 2eeffbb366..eeae652e02 100644
> --- a/meta/classes/image_types.bbclass
> +++ b/meta/classes/image_types.bbclass
> @@ -269,7 +269,7 @@ IMAGE_TYPES = " \
>  hddimg \
>  squashfs squashfs-xz squashfs-lzo squashfs-lz4 \
>  ubi ubifs multiubi \
> -tar tar.gz tar.bz2 tar.xz tar.lz4 \
> +tar tar.gz tar.bz2 tar.xz tar.lz4 tar.zst \
>  cpio cpio.gz cpio.xz cpio.lzma cpio.lz4 \
>  wic wic.gz wic.bz2 wic.lzma \
>  container \
> @@ -282,7 +282,7 @@ IMAGE_TYPES = " \
>  # CONVERSION_CMD/DEPENDS.
>  COMPRESSIONTYPES ?= ""
>
> -CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum sha224sum 
> sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2 base64 
> ${COMPRESSIONTYPES}"
> +CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip zst sum md5sum sha1sum 
> sha224sum sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2 base64 
> ${COMPRESSIONTYPES}"
>  CONVERSION_CMD_lzma = "lzma -k -f -7 
> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
>  CONVERSION_CMD_gz = "gzip -f -9 -n -c --rsyncable 
> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > 
> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
>  CONVERSION_CMD_bz2 = "pbzip2 -f -k ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
> @@ -290,6 +290,7 @@ CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} 
> ${XZ_DEFAULTS} --check=
>  CONVERSION_CMD_lz4 = "lz4 -9 -z -l ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} 
> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"
>  CONVERSION_CMD_lzo = "lzop -9 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
>  CONVERSION_CMD_zip = "zip ${ZIP_COMPRESSION_LEVEL} 
> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.zip 
> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
> +CONVERSION_CMD_zst = "zstd -f -k -c ${ZSTD_COMPRESSION_LEVEL} 
> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > 
> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.zst"
>  CONVERSION_CMD_sum = "sumtool -i ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} 
> -o ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sum ${JFFS2_SUM_EXTRA_ARGS}"
>  CONVERSION_CMD_md5sum = "md5sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > 
> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.md5sum"
>  CONVERSION_CMD_sha1sum = "sha1sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} 
> > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sha1sum"
> @@ -310,6 +311,7 @@ CONVERSION_DEPENDS_xz = "xz-native"
>  CONVERSION_DEPENDS_lz4 = "lz4-native"
>  CONVERSION_DEPENDS_lzo = "lzop-native"
>  CONVERSION_DEPENDS_zip = "zip-native"
> +CONVERSION_DEPENDS_zst = "zstd-native"
>  CONVERSION_DEPENDS_sum = "mtd-utils-native"
>  CONVERSION_DEPENDS_bmap = "bmap-tools-native"
>  CONVERSION_DEPENDS_u-boot = "u-boot-tools-native"
> --
> 2.17.1
>
> --
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core