subject:"\[PATCH 3\/4\] btrfs\: Add zstd support"

Re: [PATCH 3/4] btrfs: Add zstd support

2017-06-26 Thread Nick Terrell

Thanks for the clarification! I will fix the divisions.

On 6/26/17, 5:12 AM, "David Sterba"  wrote:

On Sun, Jun 25, 2017 at 11:30:22PM +0200, Adam Borowski wrote:
> On Mon, Jun 26, 2017 at 03:03:17AM +0800, kbuild test robot wrote:
> > Hi Nick,
> > 
> > url:
https://github.com/0day-ci/linux/commits/Nick-Terrell/lib-Add-xxhash-module/20170625-214344
> > config: i386-allmodconfig (attached as .config)
> > compiler: gcc-6 (Debian 6.2.0-3) 6.2.0 20160901
> > reproduce:
> > # save the attached .config to linux build tree
> > make ARCH=i386 
> > 
> > All errors (new ones prefixed by >>):
> > 
> > >> ERROR: "__udivdi3" [lib/zstd/zstd_compress.ko] undefined!
> >ERROR: "__udivdi3" [fs/ufs/ufs.ko] undefined!
> 
> Just to save you time to figure it out:
> for division when one or both arguments are longer than the architecture's
> word, gcc uses helper functions that are included when compiling in a 
hosted
> environment -- but not in freestanding.
> 
> Thus, you want do_div() instead of /; do check widths and signedness of
> arguments.

No do_div please, div_u64 or div64_u64.

Re: [PATCH 3/4] btrfs: Add zstd support

2017-06-26 Thread Nick Terrell

Thanks for the clarification! I will fix the divisions.

On 6/26/17, 5:12 AM, "David Sterba"  wrote:

On Sun, Jun 25, 2017 at 11:30:22PM +0200, Adam Borowski wrote:
> On Mon, Jun 26, 2017 at 03:03:17AM +0800, kbuild test robot wrote:
> > Hi Nick,
> > 
> > url:
https://github.com/0day-ci/linux/commits/Nick-Terrell/lib-Add-xxhash-module/20170625-214344
> > config: i386-allmodconfig (attached as .config)
> > compiler: gcc-6 (Debian 6.2.0-3) 6.2.0 20160901
> > reproduce:
> > # save the attached .config to linux build tree
> > make ARCH=i386 
> > 
> > All errors (new ones prefixed by >>):
> > 
> > >> ERROR: "__udivdi3" [lib/zstd/zstd_compress.ko] undefined!
> >ERROR: "__udivdi3" [fs/ufs/ufs.ko] undefined!
> 
> Just to save you time to figure it out:
> for division when one or both arguments are longer than the architecture's
> word, gcc uses helper functions that are included when compiling in a 
hosted
> environment -- but not in freestanding.
> 
> Thus, you want do_div() instead of /; do check widths and signedness of
> arguments.

No do_div please, div_u64 or div64_u64.

Re: [PATCH 3/4] btrfs: Add zstd support

2017-06-26 Thread David Sterba

On Sun, Jun 25, 2017 at 11:30:22PM +0200, Adam Borowski wrote:
> On Mon, Jun 26, 2017 at 03:03:17AM +0800, kbuild test robot wrote:
> > Hi Nick,
> > 
> > url:
> > https://github.com/0day-ci/linux/commits/Nick-Terrell/lib-Add-xxhash-module/20170625-214344
> > config: i386-allmodconfig (attached as .config)
> > compiler: gcc-6 (Debian 6.2.0-3) 6.2.0 20160901
> > reproduce:
> > # save the attached .config to linux build tree
> > make ARCH=i386 
> > 
> > All errors (new ones prefixed by >>):
> > 
> > >> ERROR: "__udivdi3" [lib/zstd/zstd_compress.ko] undefined!
> >ERROR: "__udivdi3" [fs/ufs/ufs.ko] undefined!
> 
> Just to save you time to figure it out:
> for division when one or both arguments are longer than the architecture's
> word, gcc uses helper functions that are included when compiling in a hosted
> environment -- but not in freestanding.
> 
> Thus, you want do_div() instead of /; do check widths and signedness of
> arguments.

No do_div please, div_u64 or div64_u64.

Re: [PATCH 3/4] btrfs: Add zstd support

2017-06-26 Thread David Sterba

On Sun, Jun 25, 2017 at 11:30:22PM +0200, Adam Borowski wrote:
> On Mon, Jun 26, 2017 at 03:03:17AM +0800, kbuild test robot wrote:
> > Hi Nick,
> > 
> > url:
> > https://github.com/0day-ci/linux/commits/Nick-Terrell/lib-Add-xxhash-module/20170625-214344
> > config: i386-allmodconfig (attached as .config)
> > compiler: gcc-6 (Debian 6.2.0-3) 6.2.0 20160901
> > reproduce:
> > # save the attached .config to linux build tree
> > make ARCH=i386 
> > 
> > All errors (new ones prefixed by >>):
> > 
> > >> ERROR: "__udivdi3" [lib/zstd/zstd_compress.ko] undefined!
> >ERROR: "__udivdi3" [fs/ufs/ufs.ko] undefined!
> 
> Just to save you time to figure it out:
> for division when one or both arguments are longer than the architecture's
> word, gcc uses helper functions that are included when compiling in a hosted
> environment -- but not in freestanding.
> 
> Thus, you want do_div() instead of /; do check widths and signedness of
> arguments.

No do_div please, div_u64 or div64_u64.

Re: [PATCH 3/4] btrfs: Add zstd support

2017-06-25 Thread Adam Borowski

On Mon, Jun 26, 2017 at 03:03:17AM +0800, kbuild test robot wrote:
> Hi Nick,
> 
> url:
> https://github.com/0day-ci/linux/commits/Nick-Terrell/lib-Add-xxhash-module/20170625-214344
> config: i386-allmodconfig (attached as .config)
> compiler: gcc-6 (Debian 6.2.0-3) 6.2.0 20160901
> reproduce:
> # save the attached .config to linux build tree
> make ARCH=i386 
> 
> All errors (new ones prefixed by >>):
> 
> >> ERROR: "__udivdi3" [lib/zstd/zstd_compress.ko] undefined!
>ERROR: "__udivdi3" [fs/ufs/ufs.ko] undefined!

Just to save you time to figure it out:
for division when one or both arguments are longer than the architecture's
word, gcc uses helper functions that are included when compiling in a hosted
environment -- but not in freestanding.

Thus, you want do_div() instead of /; do check widths and signedness of
arguments.

Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ 
⣾⠁⢠⠒⠀⣿⡁ A dumb species has no way to open a tuna can.
⢿⡄⠘⠷⠚⠋⠀ A smart species invents a can opener.
⠈⠳⣄ A master species delegates.

Re: [PATCH 3/4] btrfs: Add zstd support

2017-06-25 Thread Adam Borowski

On Mon, Jun 26, 2017 at 03:03:17AM +0800, kbuild test robot wrote:
> Hi Nick,
> 
> url:
> https://github.com/0day-ci/linux/commits/Nick-Terrell/lib-Add-xxhash-module/20170625-214344
> config: i386-allmodconfig (attached as .config)
> compiler: gcc-6 (Debian 6.2.0-3) 6.2.0 20160901
> reproduce:
> # save the attached .config to linux build tree
> make ARCH=i386 
> 
> All errors (new ones prefixed by >>):
> 
> >> ERROR: "__udivdi3" [lib/zstd/zstd_compress.ko] undefined!
>ERROR: "__udivdi3" [fs/ufs/ufs.ko] undefined!

Just to save you time to figure it out:
for division when one or both arguments are longer than the architecture's
word, gcc uses helper functions that are included when compiling in a hosted
environment -- but not in freestanding.

Thus, you want do_div() instead of /; do check widths and signedness of
arguments.

Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ 
⣾⠁⢠⠒⠀⣿⡁ A dumb species has no way to open a tuna can.
⢿⡄⠘⠷⠚⠋⠀ A smart species invents a can opener.
⠈⠳⣄ A master species delegates.

Re: [PATCH 3/4] btrfs: Add zstd support

2017-06-25 Thread kbuild test robot

Hi Nick,

[auto build test ERROR on linus/master]
[also build test ERROR on v4.12-rc6]
[if your patch is applied to the wrong git tree, please drop us a note to help 
improve the system]

url:
https://github.com/0day-ci/linux/commits/Nick-Terrell/lib-Add-xxhash-module/20170625-214344
config: i386-allmodconfig (attached as .config)
compiler: gcc-6 (Debian 6.2.0-3) 6.2.0 20160901
reproduce:
# save the attached .config to linux build tree
make ARCH=i386 

All errors (new ones prefixed by >>):

>> ERROR: "__udivdi3" [lib/zstd/zstd_compress.ko] undefined!
   ERROR: "__udivdi3" [fs/ufs/ufs.ko] undefined!

---
0-DAY kernel test infrastructureOpen Source Technology Center
https://lists.01.org/pipermail/kbuild-all   Intel Corporation


.config.gz
Description: application/gzip

Re: [PATCH 3/4] btrfs: Add zstd support

2017-06-25 Thread kbuild test robot

Hi Nick,

[auto build test ERROR on linus/master]
[also build test ERROR on v4.12-rc6]
[if your patch is applied to the wrong git tree, please drop us a note to help 
improve the system]

url:
https://github.com/0day-ci/linux/commits/Nick-Terrell/lib-Add-xxhash-module/20170625-214344
config: i386-allmodconfig (attached as .config)
compiler: gcc-6 (Debian 6.2.0-3) 6.2.0 20160901
reproduce:
# save the attached .config to linux build tree
make ARCH=i386 

All errors (new ones prefixed by >>):

>> ERROR: "__udivdi3" [lib/zstd/zstd_compress.ko] undefined!
   ERROR: "__udivdi3" [fs/ufs/ufs.ko] undefined!

---
0-DAY kernel test infrastructureOpen Source Technology Center
https://lists.01.org/pipermail/kbuild-all   Intel Corporation


.config.gz
Description: application/gzip

Re: [PATCH 3/4] btrfs: Add zstd support

2017-06-25 Thread kbuild test robot

Hi Nick,

[auto build test ERROR on linus/master]
[also build test ERROR on v4.12-rc6 next-20170623]
[if your patch is applied to the wrong git tree, please drop us a note to help 
improve the system]

url:
https://github.com/0day-ci/linux/commits/Nick-Terrell/lib-Add-xxhash-module/20170625-214344
config: blackfin-allyesconfig (attached as .config)
compiler: bfin-uclinux-gcc (GCC) 6.2.0
reproduce:
wget 
https://raw.githubusercontent.com/01org/lkp-tests/master/sbin/make.cross -O 
~/bin/make.cross
chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
make.cross ARCH=blackfin 

All error/warnings (new ones prefixed by >>):

   lib//zstd/fse_compress.c: In function 'FSE_buildCTable_wksp':
>> lib//zstd/fse_compress.c:181:1: warning: the frame size of 1036 bytes is 
>> larger than 1024 bytes [-Wframe-larger-than=]
}
^
   lib//zstd/fse_compress.c: In function 'FSE_compress_wksp':
   lib//zstd/fse_compress.c:857:1: warning: the frame size of 1552 bytes is 
larger than 1024 bytes [-Wframe-larger-than=]
}
^
--
   lib//zstd/compress.c: In function 'ZSTD_compressBlock_lazy':
>> lib//zstd/compress.c:2036:1: error: unable to find a register to spill in 
>> class 'CCREGS'
static void ZSTD_compressBlock_lazy(ZSTD_CCtx *ctx, const void *src, size_t 
srcSize) { ZSTD_compressBlock_lazy_generic(ctx, src, srcSize, 0, 1); }
^~
>> lib//zstd/compress.c:2036:1: error: this is the insn:
   (insn 213 11 1172 9 (set (reg:BI 1429)
   (eq:BI (reg/v:SI 62 [ mls ])
   (const_int 5 [0x5]))) lib//zstd/compress.c:1855 118 {compare_eq}
(nil))
   lib//zstd/compress.c:2036: confused by earlier errors, bailing out
--
   lib//zstd/huf_decompress.c: In function 'HUF_readDTableX4':
>> lib//zstd/huf_decompress.c:556:1: warning: the frame size of 1636 bytes is 
>> larger than 1024 bytes [-Wframe-larger-than=]
}
^

vim +/CCREGS +2036 lib//zstd/compress.c

87a5643e Nick Terrell 2017-06-22  2020  /* Save reps for next block */
87a5643e Nick Terrell 2017-06-22  2021  ctx->repToConfirm[0] = offset_1 
? offset_1 : savedOffset;
87a5643e Nick Terrell 2017-06-22  2022  ctx->repToConfirm[1] = offset_2 
? offset_2 : savedOffset;
87a5643e Nick Terrell 2017-06-22  2023  
87a5643e Nick Terrell 2017-06-22  2024  /* Last Literals */
87a5643e Nick Terrell 2017-06-22  2025  {
87a5643e Nick Terrell 2017-06-22  2026  size_t const lastLLSize 
= iend - anchor;
87a5643e Nick Terrell 2017-06-22  2027  
memcpy(seqStorePtr->lit, anchor, lastLLSize);
87a5643e Nick Terrell 2017-06-22  2028  seqStorePtr->lit += 
lastLLSize;
87a5643e Nick Terrell 2017-06-22  2029  }
87a5643e Nick Terrell 2017-06-22  2030  }
87a5643e Nick Terrell 2017-06-22  2031  
87a5643e Nick Terrell 2017-06-22  2032  static void 
ZSTD_compressBlock_btlazy2(ZSTD_CCtx *ctx, const void *src, size_t srcSize) { 
ZSTD_compressBlock_lazy_generic(ctx, src, srcSize, 1, 2); }
87a5643e Nick Terrell 2017-06-22  2033  
87a5643e Nick Terrell 2017-06-22  2034  static void 
ZSTD_compressBlock_lazy2(ZSTD_CCtx *ctx, const void *src, size_t srcSize) { 
ZSTD_compressBlock_lazy_generic(ctx, src, srcSize, 0, 2); }
87a5643e Nick Terrell 2017-06-22  2035  
87a5643e Nick Terrell 2017-06-22 @2036  static void 
ZSTD_compressBlock_lazy(ZSTD_CCtx *ctx, const void *src, size_t srcSize) { 
ZSTD_compressBlock_lazy_generic(ctx, src, srcSize, 0, 1); }
87a5643e Nick Terrell 2017-06-22  2037  
87a5643e Nick Terrell 2017-06-22  2038  static void 
ZSTD_compressBlock_greedy(ZSTD_CCtx *ctx, const void *src, size_t srcSize) { 
ZSTD_compressBlock_lazy_generic(ctx, src, srcSize, 0, 0); }
87a5643e Nick Terrell 2017-06-22  2039  
87a5643e Nick Terrell 2017-06-22  2040  FORCE_INLINE
87a5643e Nick Terrell 2017-06-22  2041  void 
ZSTD_compressBlock_lazy_extDict_generic(ZSTD_CCtx *ctx, const void *src, size_t 
srcSize, const U32 searchMethod, const U32 depth)
87a5643e Nick Terrell 2017-06-22  2042  {
87a5643e Nick Terrell 2017-06-22  2043  seqStore_t *seqStorePtr = 
&(ctx->seqStore);
87a5643e Nick Terrell 2017-06-22  2044  const BYTE *const istart = 
(const BYTE *)src;

:: The code at line 2036 was first introduced by commit
:: 87a5643e3b02e4cb9fb83bf8f6da13be18677883 lib: Add zstd modules

:: TO: Nick Terrell 
:: CC: 0day robot 

---
0-DAY kernel test infrastructureOpen Source Technology Center
https://lists.01.org/pipermail/kbuild-all   Intel Corporation


.config.gz
Description: application/gzip

Re: [PATCH 3/4] btrfs: Add zstd support

2017-06-25 Thread kbuild test robot

Hi Nick,

[auto build test ERROR on linus/master]
[also build test ERROR on v4.12-rc6 next-20170623]
[if your patch is applied to the wrong git tree, please drop us a note to help 
improve the system]

url:
https://github.com/0day-ci/linux/commits/Nick-Terrell/lib-Add-xxhash-module/20170625-214344
config: blackfin-allyesconfig (attached as .config)
compiler: bfin-uclinux-gcc (GCC) 6.2.0
reproduce:
wget 
https://raw.githubusercontent.com/01org/lkp-tests/master/sbin/make.cross -O 
~/bin/make.cross
chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
make.cross ARCH=blackfin 

All error/warnings (new ones prefixed by >>):

   lib//zstd/fse_compress.c: In function 'FSE_buildCTable_wksp':
>> lib//zstd/fse_compress.c:181:1: warning: the frame size of 1036 bytes is 
>> larger than 1024 bytes [-Wframe-larger-than=]
}
^
   lib//zstd/fse_compress.c: In function 'FSE_compress_wksp':
   lib//zstd/fse_compress.c:857:1: warning: the frame size of 1552 bytes is 
larger than 1024 bytes [-Wframe-larger-than=]
}
^
--
   lib//zstd/compress.c: In function 'ZSTD_compressBlock_lazy':
>> lib//zstd/compress.c:2036:1: error: unable to find a register to spill in 
>> class 'CCREGS'
static void ZSTD_compressBlock_lazy(ZSTD_CCtx *ctx, const void *src, size_t 
srcSize) { ZSTD_compressBlock_lazy_generic(ctx, src, srcSize, 0, 1); }
^~
>> lib//zstd/compress.c:2036:1: error: this is the insn:
   (insn 213 11 1172 9 (set (reg:BI 1429)
   (eq:BI (reg/v:SI 62 [ mls ])
   (const_int 5 [0x5]))) lib//zstd/compress.c:1855 118 {compare_eq}
(nil))
   lib//zstd/compress.c:2036: confused by earlier errors, bailing out
--
   lib//zstd/huf_decompress.c: In function 'HUF_readDTableX4':
>> lib//zstd/huf_decompress.c:556:1: warning: the frame size of 1636 bytes is 
>> larger than 1024 bytes [-Wframe-larger-than=]
}
^

vim +/CCREGS +2036 lib//zstd/compress.c

87a5643e Nick Terrell 2017-06-22  2020  /* Save reps for next block */
87a5643e Nick Terrell 2017-06-22  2021  ctx->repToConfirm[0] = offset_1 
? offset_1 : savedOffset;
87a5643e Nick Terrell 2017-06-22  2022  ctx->repToConfirm[1] = offset_2 
? offset_2 : savedOffset;
87a5643e Nick Terrell 2017-06-22  2023  
87a5643e Nick Terrell 2017-06-22  2024  /* Last Literals */
87a5643e Nick Terrell 2017-06-22  2025  {
87a5643e Nick Terrell 2017-06-22  2026  size_t const lastLLSize 
= iend - anchor;
87a5643e Nick Terrell 2017-06-22  2027  
memcpy(seqStorePtr->lit, anchor, lastLLSize);
87a5643e Nick Terrell 2017-06-22  2028  seqStorePtr->lit += 
lastLLSize;
87a5643e Nick Terrell 2017-06-22  2029  }
87a5643e Nick Terrell 2017-06-22  2030  }
87a5643e Nick Terrell 2017-06-22  2031  
87a5643e Nick Terrell 2017-06-22  2032  static void 
ZSTD_compressBlock_btlazy2(ZSTD_CCtx *ctx, const void *src, size_t srcSize) { 
ZSTD_compressBlock_lazy_generic(ctx, src, srcSize, 1, 2); }
87a5643e Nick Terrell 2017-06-22  2033  
87a5643e Nick Terrell 2017-06-22  2034  static void 
ZSTD_compressBlock_lazy2(ZSTD_CCtx *ctx, const void *src, size_t srcSize) { 
ZSTD_compressBlock_lazy_generic(ctx, src, srcSize, 0, 2); }
87a5643e Nick Terrell 2017-06-22  2035  
87a5643e Nick Terrell 2017-06-22 @2036  static void 
ZSTD_compressBlock_lazy(ZSTD_CCtx *ctx, const void *src, size_t srcSize) { 
ZSTD_compressBlock_lazy_generic(ctx, src, srcSize, 0, 1); }
87a5643e Nick Terrell 2017-06-22  2037  
87a5643e Nick Terrell 2017-06-22  2038  static void 
ZSTD_compressBlock_greedy(ZSTD_CCtx *ctx, const void *src, size_t srcSize) { 
ZSTD_compressBlock_lazy_generic(ctx, src, srcSize, 0, 0); }
87a5643e Nick Terrell 2017-06-22  2039  
87a5643e Nick Terrell 2017-06-22  2040  FORCE_INLINE
87a5643e Nick Terrell 2017-06-22  2041  void 
ZSTD_compressBlock_lazy_extDict_generic(ZSTD_CCtx *ctx, const void *src, size_t 
srcSize, const U32 searchMethod, const U32 depth)
87a5643e Nick Terrell 2017-06-22  2042  {
87a5643e Nick Terrell 2017-06-22  2043  seqStore_t *seqStorePtr = 
&(ctx->seqStore);
87a5643e Nick Terrell 2017-06-22  2044  const BYTE *const istart = 
(const BYTE *)src;

:: The code at line 2036 was first introduced by commit
:: 87a5643e3b02e4cb9fb83bf8f6da13be18677883 lib: Add zstd modules

:: TO: Nick Terrell 
:: CC: 0day robot 

---
0-DAY kernel test infrastructureOpen Source Technology Center
https://lists.01.org/pipermail/kbuild-all   Intel Corporation


.config.gz
Description: application/gzip

[PATCH 3/4] btrfs: Add zstd support

2017-06-22 Thread Nick Terrell

Add zstd compression and decompression support to BtrFS. zstd at its
fastest level compresses almost as well as zlib, while offering much
faster compression and decompression, approaching lzo speeds.

I benchmarked btrfs with zstd compression against no compression, lzo
compression, and zlib compression. I benchmarked two scenarios. Copying
a set of files to btrfs, and then reading the files. Copying a tarball
to btrfs, extracting it to btrfs, and then reading the extracted files.
After every operation, I call `sync` and include the sync time.
Between every pair of operations I unmount and remount the filesystem
to avoid caching. The benchmark files can be found in the upstream
zstd source repository under
`contrib/linux-kernel/{btrfs-benchmark.sh,btrfs-extract-benchmark.sh}`
[1] [2].

I ran the benchmarks on a Ubuntu 14.04 VM with 2 cores and 4 GiB of RAM.
The VM is running on a MacBook Pro with a 3.1 GHz Intel Core i7 processor,
16 GB of RAM, and a SSD.

The first compression benchmark is copying 10 copies of the unzipped
Silesia corpus [3] into a BtrFS filesystem mounted with
`-o compress-force=Method`. The decompression benchmark times how long
it takes to `tar` all 10 copies into `/dev/null`. The compression ratio is
measured by comparing the output of `df` and `du`. See the benchmark file
[1] for details. I benchmarked multiple zstd compression levels, although
the patch uses zstd level 1.

| Method  | Ratio | Compression MB/s | Decompression speed |
|-|---|--|-|
| None|  0.99 |  504 | 686 |
| lzo |  1.66 |  398 | 442 |
| zlib|  2.58 |   65 | 241 |
| zstd 1  |  2.57 |  260 | 383 |
| zstd 3  |  2.71 |  174 | 408 |
| zstd 6  |  2.87 |   70 | 398 |
| zstd 9  |  2.92 |   43 | 406 |
| zstd 12 |  2.93 |   21 | 408 |
| zstd 15 |  3.01 |   11 | 354 |

The next benchmark first copies `linux-4.11.6.tar` [4] to btrfs. Then it
measures the compression ratio, extracts the tar, and deletes the tar.
Then it measures the compression ratio again, and `tar`s the extracted
files into `/dev/null`. See the benchmark file [2] for details.

| Method | Tar Ratio | Extract Ratio | Copy (s) | Extract (s)| Read (s) |
||---|---|--||--|
| None   |  0.97 |  0.78 |0.981 |  5.501 |8.807 |
| lzo|  2.06 |  1.38 |1.631 |  8.458 |8.585 |
| zlib   |  3.40 |  1.86 |7.750 | 21.544 |   11.744 |
| zstd 1 |  3.57 |  1.85 |2.579 | 11.479 |9.389 |

[1] 
https://github.com/facebook/zstd/blob/dev/contrib/linux-kernel/btrfs-benchmark.sh
[2] 
https://github.com/facebook/zstd/blob/dev/contrib/linux-kernel/btrfs-extract-benchmark.sh
[3] http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia
[4] https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.11.6.tar.xz

zstd source repository: https://github.com/facebook/zstd

Signed-off-by: Nick Terrell 
---
 fs/btrfs/Kconfig   |   2 +
 fs/btrfs/Makefile  |   2 +-
 fs/btrfs/compression.c |   1 +
 fs/btrfs/compression.h |   6 +-
 fs/btrfs/ctree.h   |   1 +
 fs/btrfs/disk-io.c |   2 +
 fs/btrfs/ioctl.c   |   6 +-
 fs/btrfs/props.c   |   6 +
 fs/btrfs/super.c   |  12 +-
 fs/btrfs/sysfs.c   |   2 +
 fs/btrfs/zstd.c| 433 +
 include/uapi/linux/btrfs.h |   8 +-
 12 files changed, 469 insertions(+), 12 deletions(-)
 create mode 100644 fs/btrfs/zstd.c

diff --git a/fs/btrfs/Kconfig b/fs/btrfs/Kconfig
index 80e9c18..a26c63b 100644
--- a/fs/btrfs/Kconfig
+++ b/fs/btrfs/Kconfig
@@ -6,6 +6,8 @@ config BTRFS_FS
select ZLIB_DEFLATE
select LZO_COMPRESS
select LZO_DECOMPRESS
+   select ZSTD_COMPRESS
+   select ZSTD_DECOMPRESS
select RAID6_PQ
select XOR_BLOCKS
select SRCU
diff --git a/fs/btrfs/Makefile b/fs/btrfs/Makefile
index 128ce17..962a95a 100644
--- a/fs/btrfs/Makefile
+++ b/fs/btrfs/Makefile
@@ -6,7 +6,7 @@ btrfs-y += super.o ctree.o extent-tree.o print-tree.o 
root-tree.o dir-item.o \
   transaction.o inode.o file.o tree-defrag.o \
   extent_map.o sysfs.o struct-funcs.o xattr.o ordered-data.o \
   extent_io.o volumes.o async-thread.o ioctl.o locking.o orphan.o \
-  export.o tree-log.o free-space-cache.o zlib.o lzo.o \
+  export.o tree-log.o free-space-cache.o zlib.o lzo.o zstd.o \
   compression.o delayed-ref.o relocation.o delayed-inode.o scrub.o \
   reada.o backref.o ulist.o qgroup.o send.o dev-replace.o raid56.o \
   uuid-tree.o props.o hash.o free-space-tree.o
diff --git a/fs/btrfs/compression.c

[PATCH 3/4] btrfs: Add zstd support

2017-06-22 Thread Nick Terrell

Add zstd compression and decompression support to BtrFS. zstd at its
fastest level compresses almost as well as zlib, while offering much
faster compression and decompression, approaching lzo speeds.

I benchmarked btrfs with zstd compression against no compression, lzo
compression, and zlib compression. I benchmarked two scenarios. Copying
a set of files to btrfs, and then reading the files. Copying a tarball
to btrfs, extracting it to btrfs, and then reading the extracted files.
After every operation, I call `sync` and include the sync time.
Between every pair of operations I unmount and remount the filesystem
to avoid caching. The benchmark files can be found in the upstream
zstd source repository under
`contrib/linux-kernel/{btrfs-benchmark.sh,btrfs-extract-benchmark.sh}`
[1] [2].

I ran the benchmarks on a Ubuntu 14.04 VM with 2 cores and 4 GiB of RAM.
The VM is running on a MacBook Pro with a 3.1 GHz Intel Core i7 processor,
16 GB of RAM, and a SSD.

The first compression benchmark is copying 10 copies of the unzipped
Silesia corpus [3] into a BtrFS filesystem mounted with
`-o compress-force=Method`. The decompression benchmark times how long
it takes to `tar` all 10 copies into `/dev/null`. The compression ratio is
measured by comparing the output of `df` and `du`. See the benchmark file
[1] for details. I benchmarked multiple zstd compression levels, although
the patch uses zstd level 1.

| Method  | Ratio | Compression MB/s | Decompression speed |
|-|---|--|-|
| None|  0.99 |  504 | 686 |
| lzo |  1.66 |  398 | 442 |
| zlib|  2.58 |   65 | 241 |
| zstd 1  |  2.57 |  260 | 383 |
| zstd 3  |  2.71 |  174 | 408 |
| zstd 6  |  2.87 |   70 | 398 |
| zstd 9  |  2.92 |   43 | 406 |
| zstd 12 |  2.93 |   21 | 408 |
| zstd 15 |  3.01 |   11 | 354 |

The next benchmark first copies `linux-4.11.6.tar` [4] to btrfs. Then it
measures the compression ratio, extracts the tar, and deletes the tar.
Then it measures the compression ratio again, and `tar`s the extracted
files into `/dev/null`. See the benchmark file [2] for details.

| Method | Tar Ratio | Extract Ratio | Copy (s) | Extract (s)| Read (s) |
||---|---|--||--|
| None   |  0.97 |  0.78 |0.981 |  5.501 |8.807 |
| lzo|  2.06 |  1.38 |1.631 |  8.458 |8.585 |
| zlib   |  3.40 |  1.86 |7.750 | 21.544 |   11.744 |
| zstd 1 |  3.57 |  1.85 |2.579 | 11.479 |9.389 |

[1] 
https://github.com/facebook/zstd/blob/dev/contrib/linux-kernel/btrfs-benchmark.sh
[2] 
https://github.com/facebook/zstd/blob/dev/contrib/linux-kernel/btrfs-extract-benchmark.sh
[3] http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia
[4] https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.11.6.tar.xz

zstd source repository: https://github.com/facebook/zstd

Signed-off-by: Nick Terrell 
---
 fs/btrfs/Kconfig   |   2 +
 fs/btrfs/Makefile  |   2 +-
 fs/btrfs/compression.c |   1 +
 fs/btrfs/compression.h |   6 +-
 fs/btrfs/ctree.h   |   1 +
 fs/btrfs/disk-io.c |   2 +
 fs/btrfs/ioctl.c   |   6 +-
 fs/btrfs/props.c   |   6 +
 fs/btrfs/super.c   |  12 +-
 fs/btrfs/sysfs.c   |   2 +
 fs/btrfs/zstd.c| 433 +
 include/uapi/linux/btrfs.h |   8 +-
 12 files changed, 469 insertions(+), 12 deletions(-)
 create mode 100644 fs/btrfs/zstd.c

diff --git a/fs/btrfs/Kconfig b/fs/btrfs/Kconfig
index 80e9c18..a26c63b 100644
--- a/fs/btrfs/Kconfig
+++ b/fs/btrfs/Kconfig
@@ -6,6 +6,8 @@ config BTRFS_FS
select ZLIB_DEFLATE
select LZO_COMPRESS
select LZO_DECOMPRESS
+   select ZSTD_COMPRESS
+   select ZSTD_DECOMPRESS
select RAID6_PQ
select XOR_BLOCKS
select SRCU
diff --git a/fs/btrfs/Makefile b/fs/btrfs/Makefile
index 128ce17..962a95a 100644
--- a/fs/btrfs/Makefile
+++ b/fs/btrfs/Makefile
@@ -6,7 +6,7 @@ btrfs-y += super.o ctree.o extent-tree.o print-tree.o 
root-tree.o dir-item.o \
   transaction.o inode.o file.o tree-defrag.o \
   extent_map.o sysfs.o struct-funcs.o xattr.o ordered-data.o \
   extent_io.o volumes.o async-thread.o ioctl.o locking.o orphan.o \
-  export.o tree-log.o free-space-cache.o zlib.o lzo.o \
+  export.o tree-log.o free-space-cache.o zlib.o lzo.o zstd.o \
   compression.o delayed-ref.o relocation.o delayed-inode.o scrub.o \
   reada.o backref.o ulist.o qgroup.o send.o dev-replace.o raid56.o \
   uuid-tree.o props.o hash.o free-space-tree.o
diff --git a/fs/btrfs/compression.c

Re: [PATCH 3/4] btrfs: Add zstd support

Re: [PATCH 3/4] btrfs: Add zstd support

Re: [PATCH 3/4] btrfs: Add zstd support

Re: [PATCH 3/4] btrfs: Add zstd support

Re: [PATCH 3/4] btrfs: Add zstd support

Re: [PATCH 3/4] btrfs: Add zstd support

Re: [PATCH 3/4] btrfs: Add zstd support

Re: [PATCH 3/4] btrfs: Add zstd support

Re: [PATCH 3/4] btrfs: Add zstd support

Re: [PATCH 3/4] btrfs: Add zstd support

[PATCH 3/4] btrfs: Add zstd support

[PATCH 3/4] btrfs: Add zstd support

12 matches

Site Navigation

Mail list logo

Footer information