Re: [PATCH v3 4/4] squashfs: Add zstd support

2017-07-31 Thread Nick Terrell 
On 7/30/17, 6:50 PM, "Phillip Lougher"  wrote:
> On Thu, Jul 20, 2017 at 10:27 PM, Nick Terrell  wrote:
>> Add zstd compression and decompression support to SquashFS. zstd is a
>> great fit for SquashFS because it can compress at ratios approaching xz,
>> while decompressing twice as fast as zlib. For SquashFS in particular,
>> it can decompress as fast as lzo and lz4. It also has the flexibility
>> to turn down the compression ratio for faster compression times.
>
> Hi Nick,
>
> This patch (and none of the previous versions) is showing up on
> squashfs-de...@lists.sourceforge.net.  I also think you should have
> emailed me directly as a courtesy, as I'm the Squashfs author and
> maintainer.

Sorry about that Phillip, it was an oversight on my part. I've added you to
the CC list going forward, and have subscribed to the mailing list.

>> | Method>> | Ratio | Compression MB/s | Decompression MB/s |
>> ||---|--||
>> | gzip>>   |  2.92 |>>>   15 |128 |
>> | lzo>>>|  2.64 |>>>  9.5 |217 |
>> | lz4>>>|  2.12 |>>>   94 |218 |
>> | xz>>> |  3.43 |>>>  5.5 | 35 |
>> | xz 256 KB>  |  3.53 |>>>  5.4 | 40 |
>> | zstd 1>> |  2.71 |>>>   96 |210 |
>> | zstd 5>> |  2.93 |>>>   69 |198 |
>> | zstd 10>>|  3.01 |>>>   41 |225 |
>> | zstd 15>>|  3.13 |>>> 11.4 |224 |
>> | zstd 16 256 KB |  3.24 |>>>  8.1 |210 |
>
>
> Those numbers look good to me.
>
>>
>> This patch was written by Sean Purcell , but I will be
>> taking over the submission process.
>>
>> [1] http://releases.ubuntu.com/16.10/
>> [2] 
>> https://github.com/facebook/zstd/blob/dev/contrib/linux-kernel/squashfs-benchmark.sh
>
> I can't find your patch that adds zstd to the user-land
> Squashfs-tools.  That would be handy to do any testing :-)
>
> Phillip

I'll include the squashfs-tools patch that Sean just posted in the next
version.

Re: [PATCH v3 4/4] squashfs: Add zstd support

2017-07-31 Thread Nick Terrell 
On 7/30/17, 6:50 PM, "Phillip Lougher"  wrote:
> On Thu, Jul 20, 2017 at 10:27 PM, Nick Terrell  wrote:
>> Add zstd compression and decompression support to SquashFS. zstd is a
>> great fit for SquashFS because it can compress at ratios approaching xz,
>> while decompressing twice as fast as zlib. For SquashFS in particular,
>> it can decompress as fast as lzo and lz4. It also has the flexibility
>> to turn down the compression ratio for faster compression times.
>
> Hi Nick,
>
> This patch (and none of the previous versions) is showing up on
> squashfs-de...@lists.sourceforge.net.  I also think you should have
> emailed me directly as a courtesy, as I'm the Squashfs author and
> maintainer.

Sorry about that Phillip, it was an oversight on my part. I've added you to
the CC list going forward, and have subscribed to the mailing list.

>> | Method>> | Ratio | Compression MB/s | Decompression MB/s |
>> ||---|--||
>> | gzip>>   |  2.92 |>>>   15 |128 |
>> | lzo>>>|  2.64 |>>>  9.5 |217 |
>> | lz4>>>|  2.12 |>>>   94 |218 |
>> | xz>>> |  3.43 |>>>  5.5 | 35 |
>> | xz 256 KB>  |  3.53 |>>>  5.4 | 40 |
>> | zstd 1>> |  2.71 |>>>   96 |210 |
>> | zstd 5>> |  2.93 |>>>   69 |198 |
>> | zstd 10>>|  3.01 |>>>   41 |225 |
>> | zstd 15>>|  3.13 |>>> 11.4 |224 |
>> | zstd 16 256 KB |  3.24 |>>>  8.1 |210 |
>
>
> Those numbers look good to me.
>
>>
>> This patch was written by Sean Purcell , but I will be
>> taking over the submission process.
>>
>> [1] http://releases.ubuntu.com/16.10/
>> [2] 
>> https://github.com/facebook/zstd/blob/dev/contrib/linux-kernel/squashfs-benchmark.sh
>
> I can't find your patch that adds zstd to the user-land
> Squashfs-tools.  That would be handy to do any testing :-)
>
> Phillip

I'll include the squashfs-tools patch that Sean just posted in the next
version.

Re: [PATCH v3 4/4] squashfs: Add zstd support

2017-07-30 Thread Phillip Lougher 
On Mon, Jul 31, 2017 at 2:50 AM, Phillip Lougher
 wrote:
> On Thu, Jul 20, 2017 at 10:27 PM, Nick Terrell  wrote:
>> Add zstd compression and decompression support to SquashFS. zstd is a
>> great fit for SquashFS because it can compress at ratios approaching xz,
>> while decompressing twice as fast as zlib. For SquashFS in particular,
>> it can decompress as fast as lzo and lz4. It also has the flexibility
>> to turn down the compression ratio for faster compression times.
>
> Hi Nick,
>
> This patch (and none of the previous versions) is showing up on
> squashfs-de...@lists.sourceforge.net.  I also think you should have
> emailed me directly as a courtesy, as I'm the Squashfs author and
> maintainer.

OK, you're not subscribed to squashfs-devel    I have accepted
your previous posts, but, please subscribe.

Thanks

Phillip

>
>
>> | Method | Ratio | Compression MB/s | Decompression MB/s |
>> ||---|--||
>> | gzip   |  2.92 |   15 |128 |
>> | lzo|  2.64 |  9.5 |217 |
>> | lz4|  2.12 |   94 |218 |
>> | xz |  3.43 |  5.5 | 35 |
>> | xz 256 KB  |  3.53 |  5.4 | 40 |
>> | zstd 1 |  2.71 |   96 |210 |
>> | zstd 5 |  2.93 |   69 |198 |
>> | zstd 10|  3.01 |   41 |225 |
>> | zstd 15|  3.13 | 11.4 |224 |
>> | zstd 16 256 KB |  3.24 |  8.1 |210 |
>
>
> Those numbers look good to me.
>
>>
>> This patch was written by Sean Purcell , but I will be
>> taking over the submission process.
>>
>> [1] http://releases.ubuntu.com/16.10/
>> [2] 
>> https://github.com/facebook/zstd/blob/dev/contrib/linux-kernel/squashfs-benchmark.sh
>
> I can't find your patch that adds zstd to the user-land
> Squashfs-tools.  That would be handy to do any testing :-)
>
> Phillip
>
>>
>> zstd source repository: https://github.com/facebook/zstd
>>
>> Cc: Sean Purcell 
>> Signed-off-by: Nick Terrell 
>> ---
>>  fs/squashfs/Kconfig|  14 +
>>  fs/squashfs/Makefile   |   1 +
>>  fs/squashfs/decompressor.c |   7 +++
>>  fs/squashfs/decompressor.h |   4 ++
>>  fs/squashfs/squashfs_fs.h  |   1 +
>>  fs/squashfs/zstd_wrapper.c | 150 
>> +
>>  6 files changed, 177 insertions(+)
>>  create mode 100644 fs/squashfs/zstd_wrapper.c
>>
>> diff --git a/fs/squashfs/Kconfig b/fs/squashfs/Kconfig
>> index ffb093e..1adb334 100644
>> --- a/fs/squashfs/Kconfig
>> +++ b/fs/squashfs/Kconfig
>> @@ -165,6 +165,20 @@ config SQUASHFS_XZ
>>
>>   If unsure, say N.
>>
>> +config SQUASHFS_ZSTD
>> +   bool "Include support for ZSTD compressed file systems"
>> +   depends on SQUASHFS
>> +   select ZSTD_DECOMPRESS
>> +   help
>> + Saying Y here includes support for reading Squashfs file systems
>> + compressed with ZSTD compression.  ZSTD gives better compression 
>> than
>> + the default ZLIB compression, while using less CPU.
>> +
>> + ZSTD is not the standard compression used in Squashfs and so most
>> + file systems will be readable without selecting this option.
>> +
>> + If unsure, say N.
>> +
>>  config SQUASHFS_4K_DEVBLK_SIZE
>> bool "Use 4K device block size?"
>> depends on SQUASHFS
>> diff --git a/fs/squashfs/Makefile b/fs/squashfs/Makefile
>> index 246a6f3..6655631 100644
>> --- a/fs/squashfs/Makefile
>> +++ b/fs/squashfs/Makefile
>> @@ -15,3 +15,4 @@ squashfs-$(CONFIG_SQUASHFS_LZ4) += lz4_wrapper.o
>>  squashfs-$(CONFIG_SQUASHFS_LZO) += lzo_wrapper.o
>>  squashfs-$(CONFIG_SQUASHFS_XZ) += xz_wrapper.o
>>  squashfs-$(CONFIG_SQUASHFS_ZLIB) += zlib_wrapper.o
>> +squashfs-$(CONFIG_SQUASHFS_ZSTD) += zstd_wrapper.o
>> diff --git a/fs/squashfs/decompressor.c b/fs/squashfs/decompressor.c
>> index d2bc136..8366398 100644
>> --- a/fs/squashfs/decompressor.c
>> +++ b/fs/squashfs/decompressor.c
>> @@ -65,6 +65,12 @@ static const struct squashfs_decompressor 
>> squashfs_zlib_comp_ops = {
>>  };
>>  #endif
>>
>> +#ifndef CONFIG_SQUASHFS_ZSTD
>> +static const struct squashfs_decompressor squashfs_zstd_comp_ops = {
>> +   NULL, NULL, NULL, NULL, ZSTD_COMPRESSION, "zstd", 0
>> +};
>> +#endif
>> +
>>  static const struct squashfs_decompressor squashfs_unknown_comp_ops = {
>> NULL, NULL, NULL, NULL, 0, "unknown", 0
>>  };
>> @@ -75,6 +81,7 @@ static const struct squashfs_decompressor *decompressor[] 
>> = {
>> _lzo_comp_ops,
>> _xz_comp_ops,
>> _lzma_unsupported_comp_ops,
>> +   _zstd_comp_ops,
>> _unknown_comp_ops
>>  };
>>
>> diff --git a/fs/squashfs/decompressor.h

Re: [PATCH v3 4/4] squashfs: Add zstd support

2017-07-30 Thread Phillip Lougher 
On Mon, Jul 31, 2017 at 2:50 AM, Phillip Lougher
 wrote:
> On Thu, Jul 20, 2017 at 10:27 PM, Nick Terrell  wrote:
>> Add zstd compression and decompression support to SquashFS. zstd is a
>> great fit for SquashFS because it can compress at ratios approaching xz,
>> while decompressing twice as fast as zlib. For SquashFS in particular,
>> it can decompress as fast as lzo and lz4. It also has the flexibility
>> to turn down the compression ratio for faster compression times.
>
> Hi Nick,
>
> This patch (and none of the previous versions) is showing up on
> squashfs-de...@lists.sourceforge.net.  I also think you should have
> emailed me directly as a courtesy, as I'm the Squashfs author and
> maintainer.

OK, you're not subscribed to squashfs-devel    I have accepted
your previous posts, but, please subscribe.

Thanks

Phillip

>
>
>> | Method | Ratio | Compression MB/s | Decompression MB/s |
>> ||---|--||
>> | gzip   |  2.92 |   15 |128 |
>> | lzo|  2.64 |  9.5 |217 |
>> | lz4|  2.12 |   94 |218 |
>> | xz |  3.43 |  5.5 | 35 |
>> | xz 256 KB  |  3.53 |  5.4 | 40 |
>> | zstd 1 |  2.71 |   96 |210 |
>> | zstd 5 |  2.93 |   69 |198 |
>> | zstd 10|  3.01 |   41 |225 |
>> | zstd 15|  3.13 | 11.4 |224 |
>> | zstd 16 256 KB |  3.24 |  8.1 |210 |
>
>
> Those numbers look good to me.
>
>>
>> This patch was written by Sean Purcell , but I will be
>> taking over the submission process.
>>
>> [1] http://releases.ubuntu.com/16.10/
>> [2] 
>> https://github.com/facebook/zstd/blob/dev/contrib/linux-kernel/squashfs-benchmark.sh
>
> I can't find your patch that adds zstd to the user-land
> Squashfs-tools.  That would be handy to do any testing :-)
>
> Phillip
>
>>
>> zstd source repository: https://github.com/facebook/zstd
>>
>> Cc: Sean Purcell 
>> Signed-off-by: Nick Terrell 
>> ---
>>  fs/squashfs/Kconfig|  14 +
>>  fs/squashfs/Makefile   |   1 +
>>  fs/squashfs/decompressor.c |   7 +++
>>  fs/squashfs/decompressor.h |   4 ++
>>  fs/squashfs/squashfs_fs.h  |   1 +
>>  fs/squashfs/zstd_wrapper.c | 150 
>> +
>>  6 files changed, 177 insertions(+)
>>  create mode 100644 fs/squashfs/zstd_wrapper.c
>>
>> diff --git a/fs/squashfs/Kconfig b/fs/squashfs/Kconfig
>> index ffb093e..1adb334 100644
>> --- a/fs/squashfs/Kconfig
>> +++ b/fs/squashfs/Kconfig
>> @@ -165,6 +165,20 @@ config SQUASHFS_XZ
>>
>>   If unsure, say N.
>>
>> +config SQUASHFS_ZSTD
>> +   bool "Include support for ZSTD compressed file systems"
>> +   depends on SQUASHFS
>> +   select ZSTD_DECOMPRESS
>> +   help
>> + Saying Y here includes support for reading Squashfs file systems
>> + compressed with ZSTD compression.  ZSTD gives better compression 
>> than
>> + the default ZLIB compression, while using less CPU.
>> +
>> + ZSTD is not the standard compression used in Squashfs and so most
>> + file systems will be readable without selecting this option.
>> +
>> + If unsure, say N.
>> +
>>  config SQUASHFS_4K_DEVBLK_SIZE
>> bool "Use 4K device block size?"
>> depends on SQUASHFS
>> diff --git a/fs/squashfs/Makefile b/fs/squashfs/Makefile
>> index 246a6f3..6655631 100644
>> --- a/fs/squashfs/Makefile
>> +++ b/fs/squashfs/Makefile
>> @@ -15,3 +15,4 @@ squashfs-$(CONFIG_SQUASHFS_LZ4) += lz4_wrapper.o
>>  squashfs-$(CONFIG_SQUASHFS_LZO) += lzo_wrapper.o
>>  squashfs-$(CONFIG_SQUASHFS_XZ) += xz_wrapper.o
>>  squashfs-$(CONFIG_SQUASHFS_ZLIB) += zlib_wrapper.o
>> +squashfs-$(CONFIG_SQUASHFS_ZSTD) += zstd_wrapper.o
>> diff --git a/fs/squashfs/decompressor.c b/fs/squashfs/decompressor.c
>> index d2bc136..8366398 100644
>> --- a/fs/squashfs/decompressor.c
>> +++ b/fs/squashfs/decompressor.c
>> @@ -65,6 +65,12 @@ static const struct squashfs_decompressor 
>> squashfs_zlib_comp_ops = {
>>  };
>>  #endif
>>
>> +#ifndef CONFIG_SQUASHFS_ZSTD
>> +static const struct squashfs_decompressor squashfs_zstd_comp_ops = {
>> +   NULL, NULL, NULL, NULL, ZSTD_COMPRESSION, "zstd", 0
>> +};
>> +#endif
>> +
>>  static const struct squashfs_decompressor squashfs_unknown_comp_ops = {
>> NULL, NULL, NULL, NULL, 0, "unknown", 0
>>  };
>> @@ -75,6 +81,7 @@ static const struct squashfs_decompressor *decompressor[] 
>> = {
>> _lzo_comp_ops,
>> _xz_comp_ops,
>> _lzma_unsupported_comp_ops,
>> +   _zstd_comp_ops,
>> _unknown_comp_ops
>>  };
>>
>> diff --git a/fs/squashfs/decompressor.h b/fs/squashfs/decompressor.h
>> index a25713c..0f5a8e4 100644
>> --- a/fs/squashfs/decompressor.h
>> +++

Re: [PATCH v3 4/4] squashfs: Add zstd support

2017-07-30 Thread Phillip Lougher 
On Thu, Jul 20, 2017 at 10:27 PM, Nick Terrell  wrote:
> Add zstd compression and decompression support to SquashFS. zstd is a
> great fit for SquashFS because it can compress at ratios approaching xz,
> while decompressing twice as fast as zlib. For SquashFS in particular,
> it can decompress as fast as lzo and lz4. It also has the flexibility
> to turn down the compression ratio for faster compression times.

Hi Nick,

This patch (and none of the previous versions) is showing up on
squashfs-de...@lists.sourceforge.net.  I also think you should have
emailed me directly as a courtesy, as I'm the Squashfs author and
maintainer.


> | Method | Ratio | Compression MB/s | Decompression MB/s |
> ||---|--||
> | gzip   |  2.92 |   15 |128 |
> | lzo|  2.64 |  9.5 |217 |
> | lz4|  2.12 |   94 |218 |
> | xz |  3.43 |  5.5 | 35 |
> | xz 256 KB  |  3.53 |  5.4 | 40 |
> | zstd 1 |  2.71 |   96 |210 |
> | zstd 5 |  2.93 |   69 |198 |
> | zstd 10|  3.01 |   41 |225 |
> | zstd 15|  3.13 | 11.4 |224 |
> | zstd 16 256 KB |  3.24 |  8.1 |210 |


Those numbers look good to me.

>
> This patch was written by Sean Purcell , but I will be
> taking over the submission process.
>
> [1] http://releases.ubuntu.com/16.10/
> [2] 
> https://github.com/facebook/zstd/blob/dev/contrib/linux-kernel/squashfs-benchmark.sh

I can't find your patch that adds zstd to the user-land
Squashfs-tools.  That would be handy to do any testing :-)

Phillip

>
> zstd source repository: https://github.com/facebook/zstd
>
> Cc: Sean Purcell 
> Signed-off-by: Nick Terrell 
> ---
>  fs/squashfs/Kconfig|  14 +
>  fs/squashfs/Makefile   |   1 +
>  fs/squashfs/decompressor.c |   7 +++
>  fs/squashfs/decompressor.h |   4 ++
>  fs/squashfs/squashfs_fs.h  |   1 +
>  fs/squashfs/zstd_wrapper.c | 150 
> +
>  6 files changed, 177 insertions(+)
>  create mode 100644 fs/squashfs/zstd_wrapper.c
>
> diff --git a/fs/squashfs/Kconfig b/fs/squashfs/Kconfig
> index ffb093e..1adb334 100644
> --- a/fs/squashfs/Kconfig
> +++ b/fs/squashfs/Kconfig
> @@ -165,6 +165,20 @@ config SQUASHFS_XZ
>
>   If unsure, say N.
>
> +config SQUASHFS_ZSTD
> +   bool "Include support for ZSTD compressed file systems"
> +   depends on SQUASHFS
> +   select ZSTD_DECOMPRESS
> +   help
> + Saying Y here includes support for reading Squashfs file systems
> + compressed with ZSTD compression.  ZSTD gives better compression 
> than
> + the default ZLIB compression, while using less CPU.
> +
> + ZSTD is not the standard compression used in Squashfs and so most
> + file systems will be readable without selecting this option.
> +
> + If unsure, say N.
> +
>  config SQUASHFS_4K_DEVBLK_SIZE
> bool "Use 4K device block size?"
> depends on SQUASHFS
> diff --git a/fs/squashfs/Makefile b/fs/squashfs/Makefile
> index 246a6f3..6655631 100644
> --- a/fs/squashfs/Makefile
> +++ b/fs/squashfs/Makefile
> @@ -15,3 +15,4 @@ squashfs-$(CONFIG_SQUASHFS_LZ4) += lz4_wrapper.o
>  squashfs-$(CONFIG_SQUASHFS_LZO) += lzo_wrapper.o
>  squashfs-$(CONFIG_SQUASHFS_XZ) += xz_wrapper.o
>  squashfs-$(CONFIG_SQUASHFS_ZLIB) += zlib_wrapper.o
> +squashfs-$(CONFIG_SQUASHFS_ZSTD) += zstd_wrapper.o
> diff --git a/fs/squashfs/decompressor.c b/fs/squashfs/decompressor.c
> index d2bc136..8366398 100644
> --- a/fs/squashfs/decompressor.c
> +++ b/fs/squashfs/decompressor.c
> @@ -65,6 +65,12 @@ static const struct squashfs_decompressor 
> squashfs_zlib_comp_ops = {
>  };
>  #endif
>
> +#ifndef CONFIG_SQUASHFS_ZSTD
> +static const struct squashfs_decompressor squashfs_zstd_comp_ops = {
> +   NULL, NULL, NULL, NULL, ZSTD_COMPRESSION, "zstd", 0
> +};
> +#endif
> +
>  static const struct squashfs_decompressor squashfs_unknown_comp_ops = {
> NULL, NULL, NULL, NULL, 0, "unknown", 0
>  };
> @@ -75,6 +81,7 @@ static const struct squashfs_decompressor *decompressor[] = 
> {
> _lzo_comp_ops,
> _xz_comp_ops,
> _lzma_unsupported_comp_ops,
> +   _zstd_comp_ops,
> _unknown_comp_ops
>  };
>
> diff --git a/fs/squashfs/decompressor.h b/fs/squashfs/decompressor.h
> index a25713c..0f5a8e4 100644
> --- a/fs/squashfs/decompressor.h
> +++ b/fs/squashfs/decompressor.h
> @@ -58,4 +58,8 @@ extern const struct squashfs_decompressor 
> squashfs_lzo_comp_ops;
>  extern const struct squashfs_decompressor squashfs_zlib_comp_ops;
>  #endif
>
> +#ifdef CONFIG_SQUASHFS_ZSTD
> +extern const struct

Re: [PATCH v3 4/4] squashfs: Add zstd support

2017-07-30 Thread Phillip Lougher 
On Thu, Jul 20, 2017 at 10:27 PM, Nick Terrell  wrote:
> Add zstd compression and decompression support to SquashFS. zstd is a
> great fit for SquashFS because it can compress at ratios approaching xz,
> while decompressing twice as fast as zlib. For SquashFS in particular,
> it can decompress as fast as lzo and lz4. It also has the flexibility
> to turn down the compression ratio for faster compression times.

Hi Nick,

This patch (and none of the previous versions) is showing up on
squashfs-de...@lists.sourceforge.net.  I also think you should have
emailed me directly as a courtesy, as I'm the Squashfs author and
maintainer.


> | Method | Ratio | Compression MB/s | Decompression MB/s |
> ||---|--||
> | gzip   |  2.92 |   15 |128 |
> | lzo|  2.64 |  9.5 |217 |
> | lz4|  2.12 |   94 |218 |
> | xz |  3.43 |  5.5 | 35 |
> | xz 256 KB  |  3.53 |  5.4 | 40 |
> | zstd 1 |  2.71 |   96 |210 |
> | zstd 5 |  2.93 |   69 |198 |
> | zstd 10|  3.01 |   41 |225 |
> | zstd 15|  3.13 | 11.4 |224 |
> | zstd 16 256 KB |  3.24 |  8.1 |210 |


Those numbers look good to me.

>
> This patch was written by Sean Purcell , but I will be
> taking over the submission process.
>
> [1] http://releases.ubuntu.com/16.10/
> [2] 
> https://github.com/facebook/zstd/blob/dev/contrib/linux-kernel/squashfs-benchmark.sh

I can't find your patch that adds zstd to the user-land
Squashfs-tools.  That would be handy to do any testing :-)

Phillip

>
> zstd source repository: https://github.com/facebook/zstd
>
> Cc: Sean Purcell 
> Signed-off-by: Nick Terrell 
> ---
>  fs/squashfs/Kconfig|  14 +
>  fs/squashfs/Makefile   |   1 +
>  fs/squashfs/decompressor.c |   7 +++
>  fs/squashfs/decompressor.h |   4 ++
>  fs/squashfs/squashfs_fs.h  |   1 +
>  fs/squashfs/zstd_wrapper.c | 150 
> +
>  6 files changed, 177 insertions(+)
>  create mode 100644 fs/squashfs/zstd_wrapper.c
>
> diff --git a/fs/squashfs/Kconfig b/fs/squashfs/Kconfig
> index ffb093e..1adb334 100644
> --- a/fs/squashfs/Kconfig
> +++ b/fs/squashfs/Kconfig
> @@ -165,6 +165,20 @@ config SQUASHFS_XZ
>
>   If unsure, say N.
>
> +config SQUASHFS_ZSTD
> +   bool "Include support for ZSTD compressed file systems"
> +   depends on SQUASHFS
> +   select ZSTD_DECOMPRESS
> +   help
> + Saying Y here includes support for reading Squashfs file systems
> + compressed with ZSTD compression.  ZSTD gives better compression 
> than
> + the default ZLIB compression, while using less CPU.
> +
> + ZSTD is not the standard compression used in Squashfs and so most
> + file systems will be readable without selecting this option.
> +
> + If unsure, say N.
> +
>  config SQUASHFS_4K_DEVBLK_SIZE
> bool "Use 4K device block size?"
> depends on SQUASHFS
> diff --git a/fs/squashfs/Makefile b/fs/squashfs/Makefile
> index 246a6f3..6655631 100644
> --- a/fs/squashfs/Makefile
> +++ b/fs/squashfs/Makefile
> @@ -15,3 +15,4 @@ squashfs-$(CONFIG_SQUASHFS_LZ4) += lz4_wrapper.o
>  squashfs-$(CONFIG_SQUASHFS_LZO) += lzo_wrapper.o
>  squashfs-$(CONFIG_SQUASHFS_XZ) += xz_wrapper.o
>  squashfs-$(CONFIG_SQUASHFS_ZLIB) += zlib_wrapper.o
> +squashfs-$(CONFIG_SQUASHFS_ZSTD) += zstd_wrapper.o
> diff --git a/fs/squashfs/decompressor.c b/fs/squashfs/decompressor.c
> index d2bc136..8366398 100644
> --- a/fs/squashfs/decompressor.c
> +++ b/fs/squashfs/decompressor.c
> @@ -65,6 +65,12 @@ static const struct squashfs_decompressor 
> squashfs_zlib_comp_ops = {
>  };
>  #endif
>
> +#ifndef CONFIG_SQUASHFS_ZSTD
> +static const struct squashfs_decompressor squashfs_zstd_comp_ops = {
> +   NULL, NULL, NULL, NULL, ZSTD_COMPRESSION, "zstd", 0
> +};
> +#endif
> +
>  static const struct squashfs_decompressor squashfs_unknown_comp_ops = {
> NULL, NULL, NULL, NULL, 0, "unknown", 0
>  };
> @@ -75,6 +81,7 @@ static const struct squashfs_decompressor *decompressor[] = 
> {
> _lzo_comp_ops,
> _xz_comp_ops,
> _lzma_unsupported_comp_ops,
> +   _zstd_comp_ops,
> _unknown_comp_ops
>  };
>
> diff --git a/fs/squashfs/decompressor.h b/fs/squashfs/decompressor.h
> index a25713c..0f5a8e4 100644
> --- a/fs/squashfs/decompressor.h
> +++ b/fs/squashfs/decompressor.h
> @@ -58,4 +58,8 @@ extern const struct squashfs_decompressor 
> squashfs_lzo_comp_ops;
>  extern const struct squashfs_decompressor squashfs_zlib_comp_ops;
>  #endif
>
> +#ifdef CONFIG_SQUASHFS_ZSTD
> +extern const struct squashfs_decompressor squashfs_zstd_comp_ops;
> +#endif
> +
>  #endif
>

[PATCH v3 4/4] squashfs: Add zstd support

2017-07-20 Thread Nick Terrell 
Add zstd compression and decompression support to SquashFS. zstd is a
great fit for SquashFS because it can compress at ratios approaching xz,
while decompressing twice as fast as zlib. For SquashFS in particular,
it can decompress as fast as lzo and lz4. It also has the flexibility
to turn down the compression ratio for faster compression times.

The compression benchmark is run on the file tree from the SquashFS archive
found in ubuntu-16.10-desktop-amd64.iso [1]. It uses `mksquashfs` with the
default block size (128 KB) and and various compression algorithms/levels.
xz and zstd are also benchmarked with 256 KB blocks. The decompression
benchmark times how long it takes to `tar` the file tree into `/dev/null`.
See the benchmark file in the upstream zstd source repository located under
`contrib/linux-kernel/squashfs-benchmark.sh` [2] for details.

I ran the benchmarks on a Ubuntu 14.04 VM with 2 cores and 4 GiB of RAM.
The VM is running on a MacBook Pro with a 3.1 GHz Intel Core i7 processor,
16 GB of RAM, and a SSD.

| Method | Ratio | Compression MB/s | Decompression MB/s |
||---|--||
| gzip   |  2.92 |   15 |128 |
| lzo|  2.64 |  9.5 |217 |
| lz4|  2.12 |   94 |218 |
| xz |  3.43 |  5.5 | 35 |
| xz 256 KB  |  3.53 |  5.4 | 40 |
| zstd 1 |  2.71 |   96 |210 |
| zstd 5 |  2.93 |   69 |198 |
| zstd 10|  3.01 |   41 |225 |
| zstd 15|  3.13 | 11.4 |224 |
| zstd 16 256 KB |  3.24 |  8.1 |210 |

This patch was written by Sean Purcell , but I will be
taking over the submission process.

[1] http://releases.ubuntu.com/16.10/
[2] 
https://github.com/facebook/zstd/blob/dev/contrib/linux-kernel/squashfs-benchmark.sh

zstd source repository: https://github.com/facebook/zstd

Cc: Sean Purcell 
Signed-off-by: Nick Terrell 
---
 fs/squashfs/Kconfig|  14 +
 fs/squashfs/Makefile   |   1 +
 fs/squashfs/decompressor.c |   7 +++
 fs/squashfs/decompressor.h |   4 ++
 fs/squashfs/squashfs_fs.h  |   1 +
 fs/squashfs/zstd_wrapper.c | 150 +
 6 files changed, 177 insertions(+)
 create mode 100644 fs/squashfs/zstd_wrapper.c

diff --git a/fs/squashfs/Kconfig b/fs/squashfs/Kconfig
index ffb093e..1adb334 100644
--- a/fs/squashfs/Kconfig
+++ b/fs/squashfs/Kconfig
@@ -165,6 +165,20 @@ config SQUASHFS_XZ

  If unsure, say N.

+config SQUASHFS_ZSTD
+   bool "Include support for ZSTD compressed file systems"
+   depends on SQUASHFS
+   select ZSTD_DECOMPRESS
+   help
+ Saying Y here includes support for reading Squashfs file systems
+ compressed with ZSTD compression.  ZSTD gives better compression than
+ the default ZLIB compression, while using less CPU.
+
+ ZSTD is not the standard compression used in Squashfs and so most
+ file systems will be readable without selecting this option.
+
+ If unsure, say N.
+
 config SQUASHFS_4K_DEVBLK_SIZE
bool "Use 4K device block size?"
depends on SQUASHFS
diff --git a/fs/squashfs/Makefile b/fs/squashfs/Makefile
index 246a6f3..6655631 100644
--- a/fs/squashfs/Makefile
+++ b/fs/squashfs/Makefile
@@ -15,3 +15,4 @@ squashfs-$(CONFIG_SQUASHFS_LZ4) += lz4_wrapper.o
 squashfs-$(CONFIG_SQUASHFS_LZO) += lzo_wrapper.o
 squashfs-$(CONFIG_SQUASHFS_XZ) += xz_wrapper.o
 squashfs-$(CONFIG_SQUASHFS_ZLIB) += zlib_wrapper.o
+squashfs-$(CONFIG_SQUASHFS_ZSTD) += zstd_wrapper.o
diff --git a/fs/squashfs/decompressor.c b/fs/squashfs/decompressor.c
index d2bc136..8366398 100644
--- a/fs/squashfs/decompressor.c
+++ b/fs/squashfs/decompressor.c
@@ -65,6 +65,12 @@ static const struct squashfs_decompressor 
squashfs_zlib_comp_ops = {
 };
 #endif

+#ifndef CONFIG_SQUASHFS_ZSTD
+static const struct squashfs_decompressor squashfs_zstd_comp_ops = {
+   NULL, NULL, NULL, NULL, ZSTD_COMPRESSION, "zstd", 0
+};
+#endif
+
 static const struct squashfs_decompressor squashfs_unknown_comp_ops = {
NULL, NULL, NULL, NULL, 0, "unknown", 0
 };
@@ -75,6 +81,7 @@ static const struct squashfs_decompressor *decompressor[] = {
_lzo_comp_ops,
_xz_comp_ops,
_lzma_unsupported_comp_ops,
+   _zstd_comp_ops,
_unknown_comp_ops
 };

diff --git a/fs/squashfs/decompressor.h b/fs/squashfs/decompressor.h
index a25713c..0f5a8e4 100644
--- a/fs/squashfs/decompressor.h
+++ b/fs/squashfs/decompressor.h
@@ -58,4 +58,8 @@ extern const struct squashfs_decompressor 
squashfs_lzo_comp_ops;
 extern const struct squashfs_decompressor squashfs_zlib_comp_ops;
 #endif

+#ifdef CONFIG_SQUASHFS_ZSTD
+extern const

[PATCH v3 4/4] squashfs: Add zstd support

2017-07-20 Thread Nick Terrell 
Add zstd compression and decompression support to SquashFS. zstd is a
great fit for SquashFS because it can compress at ratios approaching xz,
while decompressing twice as fast as zlib. For SquashFS in particular,
it can decompress as fast as lzo and lz4. It also has the flexibility
to turn down the compression ratio for faster compression times.

The compression benchmark is run on the file tree from the SquashFS archive
found in ubuntu-16.10-desktop-amd64.iso [1]. It uses `mksquashfs` with the
default block size (128 KB) and and various compression algorithms/levels.
xz and zstd are also benchmarked with 256 KB blocks. The decompression
benchmark times how long it takes to `tar` the file tree into `/dev/null`.
See the benchmark file in the upstream zstd source repository located under
`contrib/linux-kernel/squashfs-benchmark.sh` [2] for details.

I ran the benchmarks on a Ubuntu 14.04 VM with 2 cores and 4 GiB of RAM.
The VM is running on a MacBook Pro with a 3.1 GHz Intel Core i7 processor,
16 GB of RAM, and a SSD.

| Method | Ratio | Compression MB/s | Decompression MB/s |
||---|--||
| gzip   |  2.92 |   15 |128 |
| lzo|  2.64 |  9.5 |217 |
| lz4|  2.12 |   94 |218 |
| xz |  3.43 |  5.5 | 35 |
| xz 256 KB  |  3.53 |  5.4 | 40 |
| zstd 1 |  2.71 |   96 |210 |
| zstd 5 |  2.93 |   69 |198 |
| zstd 10|  3.01 |   41 |225 |
| zstd 15|  3.13 | 11.4 |224 |
| zstd 16 256 KB |  3.24 |  8.1 |210 |

This patch was written by Sean Purcell , but I will be
taking over the submission process.

[1] http://releases.ubuntu.com/16.10/
[2] 
https://github.com/facebook/zstd/blob/dev/contrib/linux-kernel/squashfs-benchmark.sh

zstd source repository: https://github.com/facebook/zstd

Cc: Sean Purcell 
Signed-off-by: Nick Terrell 
---
 fs/squashfs/Kconfig|  14 +
 fs/squashfs/Makefile   |   1 +
 fs/squashfs/decompressor.c |   7 +++
 fs/squashfs/decompressor.h |   4 ++
 fs/squashfs/squashfs_fs.h  |   1 +
 fs/squashfs/zstd_wrapper.c | 150 +
 6 files changed, 177 insertions(+)
 create mode 100644 fs/squashfs/zstd_wrapper.c

diff --git a/fs/squashfs/Kconfig b/fs/squashfs/Kconfig
index ffb093e..1adb334 100644
--- a/fs/squashfs/Kconfig
+++ b/fs/squashfs/Kconfig
@@ -165,6 +165,20 @@ config SQUASHFS_XZ

  If unsure, say N.

+config SQUASHFS_ZSTD
+   bool "Include support for ZSTD compressed file systems"
+   depends on SQUASHFS
+   select ZSTD_DECOMPRESS
+   help
+ Saying Y here includes support for reading Squashfs file systems
+ compressed with ZSTD compression.  ZSTD gives better compression than
+ the default ZLIB compression, while using less CPU.
+
+ ZSTD is not the standard compression used in Squashfs and so most
+ file systems will be readable without selecting this option.
+
+ If unsure, say N.
+
 config SQUASHFS_4K_DEVBLK_SIZE
bool "Use 4K device block size?"
depends on SQUASHFS
diff --git a/fs/squashfs/Makefile b/fs/squashfs/Makefile
index 246a6f3..6655631 100644
--- a/fs/squashfs/Makefile
+++ b/fs/squashfs/Makefile
@@ -15,3 +15,4 @@ squashfs-$(CONFIG_SQUASHFS_LZ4) += lz4_wrapper.o
 squashfs-$(CONFIG_SQUASHFS_LZO) += lzo_wrapper.o
 squashfs-$(CONFIG_SQUASHFS_XZ) += xz_wrapper.o
 squashfs-$(CONFIG_SQUASHFS_ZLIB) += zlib_wrapper.o
+squashfs-$(CONFIG_SQUASHFS_ZSTD) += zstd_wrapper.o
diff --git a/fs/squashfs/decompressor.c b/fs/squashfs/decompressor.c
index d2bc136..8366398 100644
--- a/fs/squashfs/decompressor.c
+++ b/fs/squashfs/decompressor.c
@@ -65,6 +65,12 @@ static const struct squashfs_decompressor 
squashfs_zlib_comp_ops = {
 };
 #endif

+#ifndef CONFIG_SQUASHFS_ZSTD
+static const struct squashfs_decompressor squashfs_zstd_comp_ops = {
+   NULL, NULL, NULL, NULL, ZSTD_COMPRESSION, "zstd", 0
+};
+#endif
+
 static const struct squashfs_decompressor squashfs_unknown_comp_ops = {
NULL, NULL, NULL, NULL, 0, "unknown", 0
 };
@@ -75,6 +81,7 @@ static const struct squashfs_decompressor *decompressor[] = {
_lzo_comp_ops,
_xz_comp_ops,
_lzma_unsupported_comp_ops,
+   _zstd_comp_ops,
_unknown_comp_ops
 };

diff --git a/fs/squashfs/decompressor.h b/fs/squashfs/decompressor.h
index a25713c..0f5a8e4 100644
--- a/fs/squashfs/decompressor.h
+++ b/fs/squashfs/decompressor.h
@@ -58,4 +58,8 @@ extern const struct squashfs_decompressor 
squashfs_lzo_comp_ops;
 extern const struct squashfs_decompressor squashfs_zlib_comp_ops;
 #endif

+#ifdef CONFIG_SQUASHFS_ZSTD
+extern const struct squashfs_decompressor squashfs_zstd_comp_ops;