Author: cem
Date: Sat Dec 29 21:18:01 2018
New Revision: 342598
URL: https://svnweb.freebsd.org/changeset/base/342598

Log:
  Update to Zstandard 1.3.8
  
  This merge brings in a couple new files, which needed to be attached to the
  build; a new dependency on <limits.h>, which must be stubbed; and a name
  change in the Context parameter constants, from ZSTD_p_foo to ZSTD_c_foo.
  
  Significantly, it fixes a kernel build error with GCC where floating-point
  functions were included in the kernel build, by hiding them under the same
  compile-time #ifdef that already covered their invocation.  That issue was
  introduced to FreeBSD in the 1.3.7 update and tracked upstream here:
  
    https://github.com/facebook/zstd/issues/1386
  
  The full 1.3.8 release notes can be found on Github:
  
    https://github.com/facebook/zstd/releases/tag/v1.3.8
  
  Relnotes:     yes

Added:
  head/sys/contrib/zstd/lib/decompress/zstd_ddict.c
     - copied unchanged from r342590, 
vendor/zstd/dist/lib/decompress/zstd_ddict.c
  head/sys/contrib/zstd/lib/decompress/zstd_ddict.h
     - copied unchanged from r342590, 
vendor/zstd/dist/lib/decompress/zstd_ddict.h
  head/sys/contrib/zstd/lib/decompress/zstd_decompress_block.c
     - copied unchanged from r342590, 
vendor/zstd/dist/lib/decompress/zstd_decompress_block.c
  head/sys/contrib/zstd/lib/decompress/zstd_decompress_block.h
     - copied unchanged from r342590, 
vendor/zstd/dist/lib/decompress/zstd_decompress_block.h
  head/sys/contrib/zstd/lib/decompress/zstd_decompress_internal.h
     - copied unchanged from r342590, 
vendor/zstd/dist/lib/decompress/zstd_decompress_internal.h
  head/sys/contrib/zstd/lib/freebsd/limits.h
     - copied, changed from r342590, head/sys/contrib/zstd/lib/freebsd/stddef.h
  head/sys/contrib/zstd/programs/benchfn.c
     - copied unchanged from r342590, vendor/zstd/dist/programs/benchfn.c
  head/sys/contrib/zstd/programs/benchfn.h
     - copied unchanged from r342590, vendor/zstd/dist/programs/benchfn.h
  head/sys/contrib/zstd/programs/benchzstd.c
     - copied unchanged from r342590, vendor/zstd/dist/programs/benchzstd.c
  head/sys/contrib/zstd/programs/benchzstd.h
     - copied unchanged from r342590, vendor/zstd/dist/programs/benchzstd.h
  head/sys/contrib/zstd/programs/util.c
     - copied unchanged from r342590, vendor/zstd/dist/programs/util.c
  head/sys/contrib/zstd/tests/regression/
     - copied from r342590, vendor/zstd/dist/tests/regression/
Deleted:
  head/sys/contrib/zstd/NEWS
  head/sys/contrib/zstd/contrib/meson/
  head/sys/contrib/zstd/programs/bench.c
  head/sys/contrib/zstd/programs/bench.h
Modified:
  head/lib/libzstd/Makefile
  head/sys/conf/files
  head/sys/contrib/zstd/Makefile
  head/sys/contrib/zstd/README.md
  head/sys/contrib/zstd/appveyor.yml
  head/sys/contrib/zstd/contrib/pzstd/Makefile
  head/sys/contrib/zstd/contrib/seekable_format/examples/parallel_processing.c
  head/sys/contrib/zstd/contrib/seekable_format/zstdseek_compress.c
  head/sys/contrib/zstd/contrib/seekable_format/zstdseek_decompress.c
  head/sys/contrib/zstd/doc/zstd_compression_format.md
  head/sys/contrib/zstd/doc/zstd_manual.html
  head/sys/contrib/zstd/lib/BUCK
  head/sys/contrib/zstd/lib/Makefile
  head/sys/contrib/zstd/lib/README.md
  head/sys/contrib/zstd/lib/common/bitstream.h
  head/sys/contrib/zstd/lib/common/compiler.h
  head/sys/contrib/zstd/lib/common/cpu.h
  head/sys/contrib/zstd/lib/common/debug.h
  head/sys/contrib/zstd/lib/common/error_private.c
  head/sys/contrib/zstd/lib/common/fse.h
  head/sys/contrib/zstd/lib/common/huf.h
  head/sys/contrib/zstd/lib/common/pool.c
  head/sys/contrib/zstd/lib/common/zstd_common.c
  head/sys/contrib/zstd/lib/common/zstd_errors.h
  head/sys/contrib/zstd/lib/common/zstd_internal.h
  head/sys/contrib/zstd/lib/compress/fse_compress.c
  head/sys/contrib/zstd/lib/compress/hist.c
  head/sys/contrib/zstd/lib/compress/hist.h
  head/sys/contrib/zstd/lib/compress/huf_compress.c
  head/sys/contrib/zstd/lib/compress/zstd_compress.c
  head/sys/contrib/zstd/lib/compress/zstd_compress_internal.h
  head/sys/contrib/zstd/lib/compress/zstd_double_fast.c
  head/sys/contrib/zstd/lib/compress/zstd_fast.c
  head/sys/contrib/zstd/lib/compress/zstd_lazy.c
  head/sys/contrib/zstd/lib/compress/zstd_ldm.c
  head/sys/contrib/zstd/lib/compress/zstd_ldm.h
  head/sys/contrib/zstd/lib/compress/zstd_opt.c
  head/sys/contrib/zstd/lib/compress/zstd_opt.h
  head/sys/contrib/zstd/lib/compress/zstdmt_compress.c
  head/sys/contrib/zstd/lib/compress/zstdmt_compress.h
  head/sys/contrib/zstd/lib/decompress/huf_decompress.c
  head/sys/contrib/zstd/lib/decompress/zstd_decompress.c
  head/sys/contrib/zstd/lib/dictBuilder/cover.c
  head/sys/contrib/zstd/lib/dictBuilder/fastcover.c
  head/sys/contrib/zstd/lib/dictBuilder/zdict.c
  head/sys/contrib/zstd/lib/legacy/zstd_v04.c
  head/sys/contrib/zstd/lib/legacy/zstd_v05.c
  head/sys/contrib/zstd/lib/zstd.h
  head/sys/contrib/zstd/programs/Makefile
  head/sys/contrib/zstd/programs/README.md
  head/sys/contrib/zstd/programs/datagen.c
  head/sys/contrib/zstd/programs/dibio.c
  head/sys/contrib/zstd/programs/fileio.c
  head/sys/contrib/zstd/programs/fileio.h
  head/sys/contrib/zstd/programs/platform.h
  head/sys/contrib/zstd/programs/util.h
  head/sys/contrib/zstd/programs/zstd.1
  head/sys/contrib/zstd/programs/zstd.1.md
  head/sys/contrib/zstd/programs/zstdcli.c
  head/sys/contrib/zstd/programs/zstdgrep
  head/sys/contrib/zstd/programs/zstdgrep.1
  head/sys/contrib/zstd/programs/zstdless.1
  head/sys/contrib/zstd/tests/Makefile
  head/sys/contrib/zstd/tests/README.md
  head/sys/contrib/zstd/tests/datagencli.c
  head/sys/contrib/zstd/tests/decodecorpus.c
  head/sys/contrib/zstd/tests/fullbench.c
  head/sys/contrib/zstd/tests/fuzz/Makefile
  head/sys/contrib/zstd/tests/fuzz/simple_round_trip.c
  head/sys/contrib/zstd/tests/fuzz/stream_round_trip.c
  head/sys/contrib/zstd/tests/fuzz/zstd_helpers.c
  head/sys/contrib/zstd/tests/fuzzer.c
  head/sys/contrib/zstd/tests/libzstd_partial_builds.sh
  head/sys/contrib/zstd/tests/longmatch.c
  head/sys/contrib/zstd/tests/paramgrill.c
  head/sys/contrib/zstd/tests/playTests.sh
  head/sys/contrib/zstd/tests/poolTests.c
  head/sys/contrib/zstd/tests/roundTripCrash.c
  head/sys/contrib/zstd/tests/seqgen.c
  head/sys/contrib/zstd/tests/symbols.c
  head/sys/contrib/zstd/tests/zstreamtest.c
  head/sys/contrib/zstd/zlibWrapper/Makefile
  head/sys/contrib/zstd/zlibWrapper/examples/zwrapbench.c
  head/sys/contrib/zstd/zlibWrapper/zstd_zlibwrapper.c
  head/sys/kern/subr_compressor.c
  head/usr.bin/zstd/Makefile
Directory Properties:
  head/sys/contrib/zstd/   (props changed)

Modified: head/lib/libzstd/Makefile
==============================================================================
--- head/lib/libzstd/Makefile   Sat Dec 29 20:02:20 2018        (r342597)
+++ head/lib/libzstd/Makefile   Sat Dec 29 21:18:01 2018        (r342598)
@@ -13,7 +13,9 @@ SRCS= entropy_common.c \
        zstd_compress.c \
        zstdmt_compress.c \
        huf_decompress.c \
+       zstd_ddict.c \
        zstd_decompress.c \
+       zstd_decompress_block.c \
        zbuff_common.c \
        zbuff_compress.c \
        zbuff_decompress.c \

Modified: head/sys/conf/files
==============================================================================
--- head/sys/conf/files Sat Dec 29 20:02:20 2018        (r342597)
+++ head/sys/conf/files Sat Dec 29 21:18:01 2018        (r342598)
@@ -652,7 +652,9 @@ contrib/zstd/lib/compress/zstd_fast.c               
optional zstdio
 contrib/zstd/lib/compress/zstd_lazy.c          optional zstdio compile-with 
${ZSTD_C}
 contrib/zstd/lib/compress/zstd_ldm.c           optional zstdio compile-with 
${ZSTD_C}
 contrib/zstd/lib/compress/zstd_opt.c           optional zstdio compile-with 
${ZSTD_C}
+contrib/zstd/lib/decompress/zstd_ddict.c       optional zstdio compile-with 
${ZSTD_C}
 contrib/zstd/lib/decompress/zstd_decompress.c  optional zstdio compile-with 
${ZSTD_C}
+contrib/zstd/lib/decompress/zstd_decompress_block.c    optional zstdio 
compile-with ${ZSTD_C}
 contrib/zstd/lib/decompress/huf_decompress.c   optional zstdio compile-with 
${ZSTD_C}
 # Blake 2
 contrib/libb2/blake2b-ref.c    optional crypto | ipsec | ipsec_support \

Modified: head/sys/contrib/zstd/Makefile
==============================================================================
--- head/sys/contrib/zstd/Makefile      Sat Dec 29 20:02:20 2018        
(r342597)
+++ head/sys/contrib/zstd/Makefile      Sat Dec 29 21:18:01 2018        
(r342598)
@@ -64,7 +64,8 @@ zlibwrapper: lib
 
 ## test: run long-duration tests
 .PHONY: test
-test: MOREFLAGS += -g -DDEBUGLEVEL=1 -Werror
+DEBUGLEVEL ?= 1
+test: MOREFLAGS += -g -DDEBUGLEVEL=$(DEBUGLEVEL) -Werror
 test:
        MOREFLAGS="$(MOREFLAGS)" $(MAKE) -j -C $(PRGDIR) allVariants
        $(MAKE) -C $(TESTDIR) $@
@@ -129,7 +130,12 @@ ifneq (,$(filter $(shell uname),Linux Darwin GNU/kFree
 HOST_OS = POSIX
 CMAKE_PARAMS = -DZSTD_BUILD_CONTRIB:BOOL=ON -DZSTD_BUILD_STATIC:BOOL=ON 
-DZSTD_BUILD_TESTS:BOOL=ON -DZSTD_ZLIB_SUPPORT:BOOL=ON 
-DZSTD_LZMA_SUPPORT:BOOL=ON -DCMAKE_BUILD_TYPE=Release
 
-EGREP = egrep --color=never
+HAVE_COLORNEVER = $(shell echo a | egrep --color=never a > /dev/null 2> 
/dev/null && echo 1 || echo 0)
+EGREP_OPTIONS ?=
+ifeq ($HAVE_COLORNEVER, 1)
+EGREP_OPTIONS += --color=never
+endif
+EGREP = egrep $(EGREP_OPTIONS)
 
 # Print a two column output of targets and their description. To add a target 
description, put a
 # comment in the Makefile with the format "## <TARGET>: <DESCRIPTION>".  For 
example:

Modified: head/sys/contrib/zstd/README.md
==============================================================================
--- head/sys/contrib/zstd/README.md     Sat Dec 29 20:02:20 2018        
(r342597)
+++ head/sys/contrib/zstd/README.md     Sat Dec 29 21:18:01 2018        
(r342598)
@@ -9,8 +9,12 @@ and a command line utility producing and decoding `.zs
 Should your project require another programming language,
 a list of known ports and bindings is provided on [Zstandard 
homepage](http://www.zstd.net/#other-languages).
 
-Development branch status : [![Build Status][travisDevBadge]][travisLink]   
[![Build status][AppveyorDevBadge]][AppveyorLink]   [![Build 
status][CircleDevBadge]][CircleLink]
+**Development branch status:**
 
+[![Build Status][travisDevBadge]][travisLink]
+[![Build status][AppveyorDevBadge]][AppveyorLink]
+[![Build status][CircleDevBadge]][CircleLink]
+
 [travisDevBadge]: https://travis-ci.org/facebook/zstd.svg?branch=dev 
"Continuous Integration test suite"
 [travisLink]: https://travis-ci.org/facebook/zstd
 [AppveyorDevBadge]: 
https://ci.appveyor.com/api/projects/status/xt38wbdxjk5mrbem/branch/dev?svg=true
 "Windows test suite"
@@ -18,7 +22,7 @@ Development branch status : [![Build Status][travisDev
 [CircleDevBadge]: 
https://circleci.com/gh/facebook/zstd/tree/dev.svg?style=shield "Short test 
suite"
 [CircleLink]: https://circleci.com/gh/facebook/zstd
 
-### Benchmarks
+## Benchmarks
 
 For reference, several fast compression algorithms were tested and compared
 on a server running Linux Debian (`Linux version 4.14.0-3-amd64`),
@@ -42,7 +46,7 @@ on the [Silesia compression corpus].
 | snappy 1.1.4            | 2.091 |   530 MB/s |  1800 MB/s |
 | lzf 3.6 -1              | 2.077 |   400 MB/s |   860 MB/s |
 
-[zlib]:http://www.zlib.net/
+[zlib]: http://www.zlib.net/
 [LZ4]: http://www.lz4.org/
 
 Zstd can also offer stronger compression ratios at the cost of compression 
speed.
@@ -65,7 +69,7 @@ A few other algorithms can produce higher compression 
 For a larger picture including slow modes, [click on this 
link](doc/images/DCspeed5.png).
 
 
-### The case for Small Data compression
+## The case for Small Data compression
 
 Previous charts provide results applicable to typical file and stream 
scenarios (several MB). Small data comes with different perspectives.
 
@@ -89,24 +93,24 @@ Training works if there is some correlation in a famil
 Hence, deploying one dictionary per type of data will provide the greatest 
benefits.
 Dictionary gains are mostly effective in the first few KB. Then, the 
compression algorithm will gradually use previously decoded content to better 
compress the rest of the file.
 
-#### Dictionary compression How To:
+### Dictionary compression How To:
 
-1) Create the dictionary
+1. Create the dictionary
 
-`zstd --train FullPathToTrainingSet/* -o dictionaryName`
+   `zstd --train FullPathToTrainingSet/* -o dictionaryName`
 
-2) Compress with dictionary
+2. Compress with dictionary
 
-`zstd -D dictionaryName FILE`
+   `zstd -D dictionaryName FILE`
 
-3) Decompress with dictionary
+3. Decompress with dictionary
 
-`zstd -D dictionaryName --decompress FILE.zst`
+   `zstd -D dictionaryName --decompress FILE.zst`
 
 
-### Build instructions
+## Build instructions
 
-#### Makefile
+### Makefile
 
 If your system is compatible with standard `make` (or `gmake`),
 invoking `make` in root directory will generate `zstd` cli in root directory.
@@ -115,7 +119,7 @@ Other available options include:
 - `make install` : create and install zstd cli, library and man pages
 - `make check` : create and run `zstd`, tests its behavior on local platform
 
-#### cmake
+### cmake
 
 A `cmake` project generator is provided within `build/cmake`.
 It can generate Makefiles or other build scripts
@@ -123,29 +127,39 @@ to create `zstd` binary, and `libzstd` dynamic and sta
 
 By default, `CMAKE_BUILD_TYPE` is set to `Release`.
 
-#### Meson
+### Meson
 
-A Meson project is provided within `contrib/meson`.
+A Meson project is provided within [`build/meson`](build/meson). Follow
+build instructions in that directory.
 
-#### Visual Studio (Windows)
+You can also take a look at [`.travis.yml`](.travis.yml) file for an
+example about how Meson is used to build this project.
 
+Note that default build type is **release**.
+
+### Visual Studio (Windows)
+
 Going into `build` directory, you will find additional possibilities:
 - Projects for Visual Studio 2005, 2008 and 2010.
   + VS2010 project is compatible with VS2012, VS2013, VS2015 and VS2017.
 - Automated build scripts for Visual compiler by 
[@KrzysFR](https://github.com/KrzysFR), in `build/VS_scripts`,
   which will build `zstd` cli and `libzstd` library without any need to open 
Visual Studio solution.
 
+### Buck
 
-### Status
+You can build the zstd binary via buck by executing: `buck build 
programs:zstd` from the root of the repo.
+The output binary will be in `buck-out/gen/programs/`.
 
+## Status
+
 Zstandard is currently deployed within Facebook. It is used continuously to 
compress large amounts of data in multiple formats and use cases.
 Zstandard is considered safe for production environments.
 
-### License
+## License
 
 Zstandard is dual-licensed under [BSD](LICENSE) and [GPLv2](COPYING).
 
-### Contributing
+## Contributing
 
 The "dev" branch is the one where all contributions are merged before reaching 
"master".
 If you plan to propose a patch, please commit into the "dev" branch, or its 
own feature branch.

Modified: head/sys/contrib/zstd/appveyor.yml
==============================================================================
--- head/sys/contrib/zstd/appveyor.yml  Sat Dec 29 20:02:20 2018        
(r342597)
+++ head/sys/contrib/zstd/appveyor.yml  Sat Dec 29 21:18:01 2018        
(r342598)
@@ -3,6 +3,8 @@
   branches:
     only:
     - master
+    - appveyorTest
+    - /visual*/
   environment:
     matrix:
     - COMPILER: "gcc"

Modified: head/sys/contrib/zstd/contrib/pzstd/Makefile
==============================================================================
--- head/sys/contrib/zstd/contrib/pzstd/Makefile        Sat Dec 29 20:02:20 
2018        (r342597)
+++ head/sys/contrib/zstd/contrib/pzstd/Makefile        Sat Dec 29 21:18:01 
2018        (r342598)
@@ -171,7 +171,7 @@ roundtripcheck: roundtrip check
        $(TESTPROG) ./test/RoundTripTest$(EXT) $(TESTFLAGS)
 
 # Build the main binary
-pzstd$(EXT): main.o Options.o Pzstd.o SkippableFrame.o $(ZSTDDIR)/libzstd.a
+pzstd$(EXT): main.o $(PROGDIR)/util.o Options.o Pzstd.o SkippableFrame.o 
$(ZSTDDIR)/libzstd.a
        $(LD_COMMAND)
 
 # Target that depends on all the tests

Modified: 
head/sys/contrib/zstd/contrib/seekable_format/examples/parallel_processing.c
==============================================================================
--- 
head/sys/contrib/zstd/contrib/seekable_format/examples/parallel_processing.c    
    Sat Dec 29 20:02:20 2018        (r342597)
+++ 
head/sys/contrib/zstd/contrib/seekable_format/examples/parallel_processing.c    
    Sat Dec 29 21:18:01 2018        (r342598)
@@ -148,20 +148,20 @@ static void sumFile_orDie(const char* fname, int nbThr
     size_t const initResult = ZSTD_seekable_initFile(seekable, fin);
     if (ZSTD_isError(initResult)) { fprintf(stderr, "ZSTD_seekable_init() 
error : %s \n", ZSTD_getErrorName(initResult)); exit(11); }
 
-    size_t const numFrames = ZSTD_seekable_getNumFrames(seekable);
+    unsigned const numFrames = ZSTD_seekable_getNumFrames(seekable);
     struct sum_job* jobs = (struct sum_job*)malloc(numFrames * sizeof(struct 
sum_job));
 
-    size_t i;
-    for (i = 0; i < numFrames; i++) {
-        jobs[i] = (struct sum_job){ fname, 0, i, 0 };
-        POOL_add(pool, sumFrame, &jobs[i]);
+    unsigned fnb;
+    for (fnb = 0; fnb < numFrames; fnb++) {
+        jobs[fnb] = (struct sum_job){ fname, 0, fnb, 0 };
+        POOL_add(pool, sumFrame, &jobs[fnb]);
     }
 
     unsigned long long total = 0;
 
-    for (i = 0; i < numFrames; i++) {
-        while (!jobs[i].done) SLEEP(5); /* wake up every 5 milliseconds to 
check */
-        total += jobs[i].sum;
+    for (fnb = 0; fnb < numFrames; fnb++) {
+        while (!jobs[fnb].done) SLEEP(5); /* wake up every 5 milliseconds to 
check */
+        total += jobs[fnb].sum;
     }
 
     printf("Sum: %llu\n", total);

Modified: head/sys/contrib/zstd/contrib/seekable_format/zstdseek_compress.c
==============================================================================
--- head/sys/contrib/zstd/contrib/seekable_format/zstdseek_compress.c   Sat Dec 
29 20:02:20 2018        (r342597)
+++ head/sys/contrib/zstd/contrib/seekable_format/zstdseek_compress.c   Sat Dec 
29 21:18:01 2018        (r342598)
@@ -8,6 +8,8 @@
  */
 
 #include <stdlib.h>     /* malloc, free */
+#include <limits.h>     /* UINT_MAX */
+#include <assert.h>
 
 #define XXH_STATIC_LINKING_ONLY
 #define XXH_NAMESPACE ZSTD_
@@ -139,7 +141,7 @@ size_t ZSTD_seekable_freeCStream(ZSTD_seekable_CStream
 size_t ZSTD_seekable_initCStream(ZSTD_seekable_CStream* zcs,
                                  int compressionLevel,
                                  int checksumFlag,
-                                 U32 maxFrameSize)
+                                 unsigned maxFrameSize)
 {
     zcs->framelog.size = 0;
     zcs->frameCSize = 0;
@@ -167,9 +169,9 @@ size_t ZSTD_seekable_initCStream(ZSTD_seekable_CStream
 }
 
 size_t ZSTD_seekable_logFrame(ZSTD_frameLog* fl,
-                                     unsigned compressedSize,
-                                     unsigned decompressedSize,
-                                     unsigned checksum)
+                              unsigned compressedSize,
+                              unsigned decompressedSize,
+                              unsigned checksum)
 {
     if (fl->size == ZSTD_SEEKABLE_MAXFRAMES)
         return ERROR(frameIndex_tooLarge);
@@ -184,7 +186,8 @@ size_t ZSTD_seekable_logFrame(ZSTD_frameLog* fl,
         if (newEntries == NULL) return ERROR(memory_allocation);
 
         fl->entries = newEntries;
-        fl->capacity = newCapacity;
+        assert(newCapacity <= UINT_MAX);
+        fl->capacity = (U32)newCapacity;
     }
 
     fl->entries[fl->size] = (framelogEntry_t){
@@ -268,7 +271,7 @@ size_t ZSTD_seekable_compressStream(ZSTD_seekable_CStr
 static inline size_t ZSTD_seekable_seekTableSize(const ZSTD_frameLog* fl)
 {
     size_t const sizePerFrame = 8 + (fl->checksumFlag?4:0);
-    size_t const seekTableLen = ZSTD_skippableHeaderSize +
+    size_t const seekTableLen = ZSTD_SKIPPABLEHEADERSIZE +
                                 sizePerFrame * fl->size +
                                 ZSTD_seekTableFooterSize;
 
@@ -307,32 +310,32 @@ size_t ZSTD_seekable_writeSeekTable(ZSTD_frameLog* fl,
     size_t const seekTableLen = ZSTD_seekable_seekTableSize(fl);
 
     CHECK_Z(ZSTD_stwrite32(fl, output, ZSTD_MAGIC_SKIPPABLE_START | 0xE, 0));
-    CHECK_Z(ZSTD_stwrite32(fl, output, seekTableLen - ZSTD_skippableHeaderSize,
-                           4));
+    assert(seekTableLen <= (size_t)UINT_MAX);
+    CHECK_Z(ZSTD_stwrite32(fl, output, (U32)seekTableLen - 
ZSTD_SKIPPABLEHEADERSIZE, 4));
 
     while (fl->seekTableIndex < fl->size) {
+        unsigned long long const start = ZSTD_SKIPPABLEHEADERSIZE + 
sizePerFrame * fl->seekTableIndex;
+        assert(start + 8 <= UINT_MAX);
         CHECK_Z(ZSTD_stwrite32(fl, output,
                                fl->entries[fl->seekTableIndex].cSize,
-                               ZSTD_skippableHeaderSize +
-                                       sizePerFrame * fl->seekTableIndex + 0));
+                               (U32)start + 0));
 
         CHECK_Z(ZSTD_stwrite32(fl, output,
                                fl->entries[fl->seekTableIndex].dSize,
-                               ZSTD_skippableHeaderSize +
-                                       sizePerFrame * fl->seekTableIndex + 4));
+                               (U32)start + 4));
 
         if (fl->checksumFlag) {
             CHECK_Z(ZSTD_stwrite32(
                     fl, output, fl->entries[fl->seekTableIndex].checksum,
-                    ZSTD_skippableHeaderSize +
-                            sizePerFrame * fl->seekTableIndex + 8));
+                    (U32)start + 8));
         }
 
         fl->seekTableIndex++;
     }
 
+    assert(seekTableLen <= UINT_MAX);
     CHECK_Z(ZSTD_stwrite32(fl, output, fl->size,
-                           seekTableLen - ZSTD_seekTableFooterSize));
+                           (U32)seekTableLen - ZSTD_seekTableFooterSize));
 
     if (output->size - output->pos < 1) return seekTableLen - fl->seekTablePos;
     if (fl->seekTablePos < seekTableLen - 4) {
@@ -345,7 +348,7 @@ size_t ZSTD_seekable_writeSeekTable(ZSTD_frameLog* fl,
     }
 
     CHECK_Z(ZSTD_stwrite32(fl, output, ZSTD_SEEKABLE_MAGICNUMBER,
-                           seekTableLen - 4));
+                           (U32)seekTableLen - 4));
 
     if (fl->seekTablePos != seekTableLen) return ERROR(GENERIC);
     return 0;

Modified: head/sys/contrib/zstd/contrib/seekable_format/zstdseek_decompress.c
==============================================================================
--- head/sys/contrib/zstd/contrib/seekable_format/zstdseek_decompress.c Sat Dec 
29 20:02:20 2018        (r342597)
+++ head/sys/contrib/zstd/contrib/seekable_format/zstdseek_decompress.c Sat Dec 
29 21:18:01 2018        (r342598)
@@ -54,8 +54,9 @@
 #   define LONG_SEEK fseek
 #endif
 
-#include <stdlib.h> /* malloc, free */
-#include <stdio.h>  /* FILE* */
+#include <stdlib.h>  /* malloc, free */
+#include <stdio.h>   /* FILE* */
+#include <limits.h>  /* UNIT_MAX */
 #include <assert.h>
 
 #define XXH_STATIC_LINKING_ONLY
@@ -200,13 +201,14 @@ size_t ZSTD_seekable_free(ZSTD_seekable* zs)
  *  Performs a binary search to find the last frame with a decompressed offset
  *  <= pos
  *  @return : the frame's index */
-U32 ZSTD_seekable_offsetToFrameIndex(ZSTD_seekable* const zs, unsigned long 
long pos)
+unsigned ZSTD_seekable_offsetToFrameIndex(ZSTD_seekable* const zs, unsigned 
long long pos)
 {
     U32 lo = 0;
-    U32 hi = zs->seekTable.tableLen;
+    U32 hi = (U32)zs->seekTable.tableLen;
+    assert(zs->seekTable.tableLen <= UINT_MAX);
 
     if (pos >= zs->seekTable.entries[zs->seekTable.tableLen].dOffset) {
-        return zs->seekTable.tableLen;
+        return (U32)zs->seekTable.tableLen;
     }
 
     while (lo + 1 < hi) {
@@ -220,31 +222,32 @@ U32 ZSTD_seekable_offsetToFrameIndex(ZSTD_seekable* co
     return lo;
 }
 
-U32 ZSTD_seekable_getNumFrames(ZSTD_seekable* const zs)
+unsigned ZSTD_seekable_getNumFrames(ZSTD_seekable* const zs)
 {
-    return zs->seekTable.tableLen;
+    assert(zs->seekTable.tableLen <= UINT_MAX);
+    return (unsigned)zs->seekTable.tableLen;
 }
 
-unsigned long long ZSTD_seekable_getFrameCompressedOffset(ZSTD_seekable* const 
zs, U32 frameIndex)
+unsigned long long ZSTD_seekable_getFrameCompressedOffset(ZSTD_seekable* const 
zs, unsigned frameIndex)
 {
     if (frameIndex >= zs->seekTable.tableLen) return 
ZSTD_SEEKABLE_FRAMEINDEX_TOOLARGE;
     return zs->seekTable.entries[frameIndex].cOffset;
 }
 
-unsigned long long ZSTD_seekable_getFrameDecompressedOffset(ZSTD_seekable* 
const zs, U32 frameIndex)
+unsigned long long ZSTD_seekable_getFrameDecompressedOffset(ZSTD_seekable* 
const zs, unsigned frameIndex)
 {
     if (frameIndex >= zs->seekTable.tableLen) return 
ZSTD_SEEKABLE_FRAMEINDEX_TOOLARGE;
     return zs->seekTable.entries[frameIndex].dOffset;
 }
 
-size_t ZSTD_seekable_getFrameCompressedSize(ZSTD_seekable* const zs, U32 
frameIndex)
+size_t ZSTD_seekable_getFrameCompressedSize(ZSTD_seekable* const zs, unsigned 
frameIndex)
 {
     if (frameIndex >= zs->seekTable.tableLen) return 
ERROR(frameIndex_tooLarge);
     return zs->seekTable.entries[frameIndex + 1].cOffset -
            zs->seekTable.entries[frameIndex].cOffset;
 }
 
-size_t ZSTD_seekable_getFrameDecompressedSize(ZSTD_seekable* const zs, U32 
frameIndex)
+size_t ZSTD_seekable_getFrameDecompressedSize(ZSTD_seekable* const zs, 
unsigned frameIndex)
 {
     if (frameIndex > zs->seekTable.tableLen) return ERROR(frameIndex_tooLarge);
     return zs->seekTable.entries[frameIndex + 1].dOffset -
@@ -275,7 +278,7 @@ static size_t ZSTD_seekable_loadSeekTable(ZSTD_seekabl
     {   U32 const numFrames = MEM_readLE32(zs->inBuff);
         U32 const sizePerEntry = 8 + (checksumFlag?4:0);
         U32 const tableSize = sizePerEntry * numFrames;
-        U32 const frameSize = tableSize + ZSTD_seekTableFooterSize + 
ZSTD_skippableHeaderSize;
+        U32 const frameSize = tableSize + ZSTD_seekTableFooterSize + 
ZSTD_SKIPPABLEHEADERSIZE;
 
         U32 remaining = frameSize - ZSTD_seekTableFooterSize; /* don't need to 
re-read footer */
         {
@@ -290,7 +293,7 @@ static size_t ZSTD_seekable_loadSeekTable(ZSTD_seekabl
         if (MEM_readLE32(zs->inBuff) != (ZSTD_MAGIC_SKIPPABLE_START | 0xE)) {
             return ERROR(prefix_unknown);
         }
-        if (MEM_readLE32(zs->inBuff+4) + ZSTD_skippableHeaderSize != 
frameSize) {
+        if (MEM_readLE32(zs->inBuff+4) + ZSTD_SKIPPABLEHEADERSIZE != 
frameSize) {
             return ERROR(prefix_unknown);
         }
 
@@ -444,7 +447,7 @@ size_t ZSTD_seekable_decompress(ZSTD_seekable* zs, voi
     return len;
 }
 
-size_t ZSTD_seekable_decompressFrame(ZSTD_seekable* zs, void* dst, size_t 
dstSize, U32 frameIndex)
+size_t ZSTD_seekable_decompressFrame(ZSTD_seekable* zs, void* dst, size_t 
dstSize, unsigned frameIndex)
 {
     if (frameIndex >= zs->seekTable.tableLen) {
         return ERROR(frameIndex_tooLarge);

Modified: head/sys/contrib/zstd/doc/zstd_compression_format.md
==============================================================================
--- head/sys/contrib/zstd/doc/zstd_compression_format.md        Sat Dec 29 
20:02:20 2018        (r342597)
+++ head/sys/contrib/zstd/doc/zstd_compression_format.md        Sat Dec 29 
21:18:01 2018        (r342598)
@@ -16,7 +16,7 @@ Distribution of this document is unlimited.
 
 ### Version
 
-0.3.0 (25/09/18)
+0.3.1 (25/10/18)
 
 
 Introduction
@@ -913,13 +913,39 @@ Note that blocks which are not `Compressed_Block` are 
 ###### Offset updates rules
 
 The newest offset takes the lead in offset history,
-shifting others back (up to its previous place if it was already present).
+shifting others back by one rank,
+up to the previous rank of the new offset _if it was present in history_.
 
-This means that when `Repeated_Offset1` (most recent) is used, history is 
unmodified.
-When `Repeated_Offset2` is used, it's swapped with `Repeated_Offset1`.
-If any other offset is used, it becomes `Repeated_Offset1` and the rest are 
shift back by one.
+__Examples__ :
 
+In the common case, when new offset is not part of history :
+`Repeated_Offset3` = `Repeated_Offset2`
+`Repeated_Offset2` = `Repeated_Offset1`
+`Repeated_Offset1` = `NewOffset`
 
+When the new offset _is_ part of history, there may be specific adjustments.
+
+When `NewOffset` == `Repeated_Offset1`, offset history remains actually 
unmodified.
+
+When `NewOffset` == `Repeated_Offset2`,
+`Repeated_Offset1` and `Repeated_Offset2` ranks are swapped.
+`Repeated_Offset3` is unmodified.
+
+When `NewOffset` == `Repeated_Offset3`,
+there is actually no difference with the common case :
+all offsets are shifted by one rank,
+`NewOffset` (== `Repeated_Offset3`) becomes the new `Repeated_Offset1`.
+
+Also worth mentioning, the specific corner case when `offset_value` == 3,
+and the literal length of the current sequence is zero.
+In which case , `NewOffset` = `Repeated_Offset1` - 1_byte.
+Here also, from an offset history update perspective, it's just a common case :
+`Repeated_Offset3` = `Repeated_Offset2`
+`Repeated_Offset2` = `Repeated_Offset1`
+`Repeated_Offset1` = `NewOffset` ( == `Repeated_Offset1` - 1_byte )
+
+
+
 Skippable Frames
 ----------------
 
@@ -1629,6 +1655,7 @@ or at least provide a meaningful error code explaining
 
 Version changes
 ---------------
+- 0.3.1 : minor clarification regarding offset history update rules
 - 0.3.0 : minor edits to match RFC8478
 - 0.2.9 : clarifications for huffman weights direct representation, by Ulrich 
Kunitz
 - 0.2.8 : clarifications for IETF RFC discuss

Modified: head/sys/contrib/zstd/doc/zstd_manual.html
==============================================================================
--- head/sys/contrib/zstd/doc/zstd_manual.html  Sat Dec 29 20:02:20 2018        
(r342597)
+++ head/sys/contrib/zstd/doc/zstd_manual.html  Sat Dec 29 21:18:01 2018        
(r342598)
@@ -1,10 +1,10 @@
 <html>
 <head>
 <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
-<title>zstd 1.3.7 Manual</title>
+<title>zstd 1.3.8 Manual</title>
 </head>
 <body>
-<h1>zstd 1.3.7 Manual</h1>
+<h1>zstd 1.3.8 Manual</h1>
 <hr>
 <a name="Contents"></a><h2>Contents</h2>
 <ol>
@@ -19,16 +19,19 @@
 <li><a href="#Chapter9">Streaming compression - HowTo</a></li>
 <li><a href="#Chapter10">Streaming decompression - HowTo</a></li>
 <li><a href="#Chapter11">ADVANCED AND EXPERIMENTAL FUNCTIONS</a></li>
-<li><a href="#Chapter12">Frame size functions</a></li>
-<li><a href="#Chapter13">Memory management</a></li>
-<li><a href="#Chapter14">Advanced compression functions</a></li>
-<li><a href="#Chapter15">Advanced decompression functions</a></li>
-<li><a href="#Chapter16">Advanced streaming functions</a></li>
-<li><a href="#Chapter17">Buffer-less and synchronous inner streaming 
functions</a></li>
-<li><a href="#Chapter18">Buffer-less streaming compression (synchronous 
mode)</a></li>
-<li><a href="#Chapter19">Buffer-less streaming decompression (synchronous 
mode)</a></li>
-<li><a href="#Chapter20">New advanced API (experimental)</a></li>
-<li><a href="#Chapter21">Block level API</a></li>
+<li><a href="#Chapter12">Candidate API for promotion to stable status</a></li>
+<li><a href="#Chapter13">Advanced compression API</a></li>
+<li><a href="#Chapter14">experimental API (static linking only)</a></li>
+<li><a href="#Chapter15">Frame size functions</a></li>
+<li><a href="#Chapter16">Memory management</a></li>
+<li><a href="#Chapter17">Advanced compression functions</a></li>
+<li><a href="#Chapter18">Advanced decompression functions</a></li>
+<li><a href="#Chapter19">Advanced streaming functions</a></li>
+<li><a href="#Chapter20">Buffer-less and synchronous inner streaming 
functions</a></li>
+<li><a href="#Chapter21">Buffer-less streaming compression (synchronous 
mode)</a></li>
+<li><a href="#Chapter22">Buffer-less streaming decompression (synchronous 
mode)</a></li>
+<li><a href="#Chapter23">ZSTD_getFrameHeader() :</a></li>
+<li><a href="#Chapter24">Block level API</a></li>
 </ol>
 <hr>
 <a name="Chapter1"></a><h2>Introduction</h2><pre>
@@ -64,7 +67,7 @@
 
 <a name="Chapter2"></a><h2>Version</h2><pre></pre>
 
-<pre><b>unsigned ZSTD_versionNumber(void);   </b>/**< useful to check dll 
version */<b>
+<pre><b>unsigned ZSTD_versionNumber(void);   </b>/**< to check runtime library 
version */<b>
 </b></pre><BR>
 <a name="Chapter3"></a><h2>Default constant</h2><pre></pre>
 
@@ -139,11 +142,13 @@ int         ZSTD_maxCLevel(void);               </b>/*
 ZSTD_CCtx* ZSTD_createCCtx(void);
 size_t     ZSTD_freeCCtx(ZSTD_CCtx* cctx);
 </pre></b><BR>
-<pre><b>size_t ZSTD_compressCCtx(ZSTD_CCtx* ctx,
+<pre><b>size_t ZSTD_compressCCtx(ZSTD_CCtx* cctx,
                          void* dst, size_t dstCapacity,
                    const void* src, size_t srcSize,
                          int compressionLevel);
-</b><p>  Same as ZSTD_compress(), requires an allocated ZSTD_CCtx (see 
ZSTD_createCCtx()). 
+</b><p>  Same as ZSTD_compress(), using an explicit ZSTD_CCtx
+  The function will compress at requested compression level,
+  ignoring any other parameter 
 </p></pre><BR>
 
 <h3>Decompression context</h3><pre>  When decompressing many times,
@@ -155,10 +160,13 @@ size_t     ZSTD_freeCCtx(ZSTD_CCtx* cctx);
 ZSTD_DCtx* ZSTD_createDCtx(void);
 size_t     ZSTD_freeDCtx(ZSTD_DCtx* dctx);
 </pre></b><BR>
-<pre><b>size_t ZSTD_decompressDCtx(ZSTD_DCtx* ctx,
+<pre><b>size_t ZSTD_decompressDCtx(ZSTD_DCtx* dctx,
                            void* dst, size_t dstCapacity,
                      const void* src, size_t srcSize);
-</b><p>  Same as ZSTD_decompress(), requires an allocated ZSTD_DCtx (see 
ZSTD_createDCtx()) 
+</b><p>  Same as ZSTD_decompress(),
+  requires an allocated ZSTD_DCtx.
+  Compatible with sticky parameters.
+ 
 </p></pre><BR>
 
 <a name="Chapter6"></a><h2>Simple dictionary API</h2><pre></pre>
@@ -168,18 +176,22 @@ size_t     ZSTD_freeDCtx(ZSTD_DCtx* dctx);
                          const void* src, size_t srcSize,
                          const void* dict,size_t dictSize,
                                int compressionLevel);
-</b><p>  Compression using a predefined Dictionary (see dictBuilder/zdict.h).
+</b><p>  Compression at an explicit compression level using a Dictionary.
+  A dictionary can be any arbitrary data segment (also called a prefix),
+  or a buffer with specified information (see dictBuilder/zdict.h).
   Note : This function loads the dictionary, resulting in significant startup 
delay.
-  Note : When `dict == NULL || dictSize < 8` no dictionary is used. 
+         It's intended for a dictionary used only once.
+  Note 2 : When `dict == NULL || dictSize < 8` no dictionary is used. 
 </p></pre><BR>
 
 <pre><b>size_t ZSTD_decompress_usingDict(ZSTD_DCtx* dctx,
                                  void* dst, size_t dstCapacity,
                            const void* src, size_t srcSize,
                            const void* dict,size_t dictSize);
-</b><p>  Decompression using a predefined Dictionary (see dictBuilder/zdict.h).
+</b><p>  Decompression using a known Dictionary.
   Dictionary must be identical to the one used during compression.
   Note : This function loads the dictionary, resulting in significant startup 
delay.
+         It's intended for a dictionary used only once.
   Note : When `dict == NULL || dictSize < 8` no dictionary is used. 
 </p></pre><BR>
 
@@ -187,11 +199,12 @@ size_t     ZSTD_freeDCtx(ZSTD_DCtx* dctx);
 
 <pre><b>ZSTD_CDict* ZSTD_createCDict(const void* dictBuffer, size_t dictSize,
                              int compressionLevel);
-</b><p>  When compressing multiple messages / blocks with the same dictionary, 
it's recommended to load it just once.
-  ZSTD_createCDict() will create a digested dictionary, ready to start future 
compression operations without startup delay.
+</b><p>  When compressing multiple messages / blocks using the same 
dictionary, it's recommended to load it only once.
+  ZSTD_createCDict() will create a digested dictionary, ready to start future 
compression operations without startup cost.
   ZSTD_CDict can be created once and shared by multiple threads concurrently, 
since its usage is read-only.
-  `dictBuffer` can be released after ZSTD_CDict creation, since its content is 
copied within CDict
-  Note : A ZSTD_CDict can be created with an empty dictionary, but it is 
inefficient for small data. 
+ `dictBuffer` can be released after ZSTD_CDict creation, because its content 
is copied within CDict.
+  Consider experimental function `ZSTD_createCDict_byReference()` if you 
prefer to not duplicate `dictBuffer` content.
+  Note : A ZSTD_CDict can be created from an empty dictBuffer, but it is 
inefficient when used to compress small data. 
 </p></pre><BR>
 
 <pre><b>size_t      ZSTD_freeCDict(ZSTD_CDict* CDict);
@@ -203,16 +216,14 @@ size_t     ZSTD_freeDCtx(ZSTD_DCtx* dctx);
                           const void* src, size_t srcSize,
                           const ZSTD_CDict* cdict);
 </b><p>  Compression using a digested Dictionary.
-  Faster startup than ZSTD_compress_usingDict(), recommended when same 
dictionary is used multiple times.
-  Note that compression level is decided during dictionary creation.
-  Frame parameters are hardcoded (dictID=yes, contentSize=yes, checksum=no)
-  Note : ZSTD_compress_usingCDict() can be used with a ZSTD_CDict created from 
an empty dictionary.
-         But it is inefficient for small data, and it is recommended to use 
ZSTD_compressCCtx(). 
+  Recommended when same dictionary is used multiple times.
+  Note : compression level is _decided at dictionary creation time_,
+     and frame parameters are hardcoded (dictID=yes, contentSize=yes, 
checksum=no) 
 </p></pre><BR>
 
 <pre><b>ZSTD_DDict* ZSTD_createDDict(const void* dictBuffer, size_t dictSize);
 </b><p>  Create a digested dictionary, ready to start decompression operation 
without startup delay.
-  dictBuffer can be released after DDict creation, as its content is copied 
inside DDict 
+  dictBuffer can be released after DDict creation, as its content is copied 
inside DDict. 
 </p></pre><BR>
 
 <pre><b>size_t      ZSTD_freeDDict(ZSTD_DDict* ddict);
@@ -224,7 +235,7 @@ size_t     ZSTD_freeDCtx(ZSTD_DCtx* dctx);
                             const void* src, size_t srcSize,
                             const ZSTD_DDict* ddict);
 </b><p>  Decompression using a digested Dictionary.
-  Faster startup than ZSTD_decompress_usingDict(), recommended when same 
dictionary is used multiple times. 
+  Recommended when same dictionary is used multiple times. 
 </p></pre><BR>
 
 <a name="Chapter8"></a><h2>Streaming</h2><pre></pre>
@@ -245,14 +256,18 @@ size_t     ZSTD_freeDCtx(ZSTD_DCtx* dctx);
   A ZSTD_CStream object is required to track streaming operation.
   Use ZSTD_createCStream() and ZSTD_freeCStream() to create/release resources.
   ZSTD_CStream objects can be reused multiple times on consecutive compression 
operations.
-  It is recommended to re-use ZSTD_CStream in situations where many streaming 
operations will be achieved consecutively,
-  since it will play nicer with system's memory, by re-using already allocated 
memory.
-  Use one separate ZSTD_CStream per thread for parallel execution.
+  It is recommended to re-use ZSTD_CStream since it will play nicer with 
system's memory, by re-using already allocated memory.
 
-  Start a new compression by initializing ZSTD_CStream context.
-  Use ZSTD_initCStream() to start a new compression operation.
-  Use variants ZSTD_initCStream_usingDict() or ZSTD_initCStream_usingCDict() 
for streaming with dictionary (experimental section)
+  For parallel execution, use one separate ZSTD_CStream per thread.
 
+  note : since v1.3.0, ZSTD_CStream and ZSTD_CCtx are the same thing.
+
+  Parameters are sticky : when starting a new compression on the same context,
+  it will re-use the same sticky parameters as previous compression session.
+  When in doubt, it's recommended to fully initialize the context before usage.
+  Use ZSTD_initCStream() to set the parameter to a selected compression level.
+  Use advanced API (ZSTD_CCtx_setParameter(), etc.) to set more specific 
parameters.
+
   Use ZSTD_compressStream() as many times as necessary to consume input stream.
   The function will automatically update both `pos` fields within `input` and 
`output`.
   Note that the function may not consume the entire input,
@@ -260,12 +275,11 @@ size_t     ZSTD_freeDCtx(ZSTD_DCtx* dctx);
   in which case `input.pos < input.size`.
   The caller must check if input has been entirely consumed.
   If not, the caller must make some room to receive more compressed data,
-  typically by emptying output buffer, or allocating a new output buffer,
   and then present again remaining input data.
-  @return : a size hint, preferred nb of bytes to use as input for next 
function call
-            or an error code, which can be tested using ZSTD_isError().
-            Note 1 : it's just a hint, to help latency a little, any other 
value will work fine.
-            Note 2 : size hint is guaranteed to be <= ZSTD_CStreamInSize()
+ @return : a size hint, preferred nb of bytes to use as input for next 
function call
+           or an error code, which can be tested using ZSTD_isError().
+           Note 1 : it's just a hint, to help latency a little, any value will 
work fine.
+           Note 2 : size hint is guaranteed to be <= ZSTD_CStreamInSize()
 
   At any moment, it's possible to flush whatever data might remain stuck 
within internal buffer,
   using ZSTD_flushStream(). `output->pos` will be updated.
@@ -305,25 +319,24 @@ size_t ZSTD_endStream(ZSTD_CStream* zcs, ZSTD_outBuffe
   Use ZSTD_createDStream() and ZSTD_freeDStream() to create/release resources.
   ZSTD_DStream objects can be re-used multiple times.
 
-  Use ZSTD_initDStream() to start a new decompression operation,
-   or ZSTD_initDStream_usingDict() if decompression requires a dictionary.
-   @return : recommended first input size
+  Use ZSTD_initDStream() to start a new decompression operation.
+ @return : recommended first input size
+  Alternatively, use advanced API to set specific properties.
 
   Use ZSTD_decompressStream() repetitively to consume your input.
   The function will update both `pos` fields.
   If `input.pos < input.size`, some input has not been consumed.
   It's up to the caller to present again remaining data.
-  The function tries to flush all data decoded immediately, repecting buffer 
sizes.
+  The function tries to flush all data decoded immediately, respecting output 
buffer size.
   If `output.pos < output.size`, decoder has flushed everything it could.
-  But if `output.pos == output.size`, there is no such guarantee,
-  it's likely that some decoded data was not flushed and still remains within 
internal buffers.
+  But if `output.pos == output.size`, there might be some data left within 
internal buffers.,
   In which case, call ZSTD_decompressStream() again to flush whatever remains 
in the buffer.
-  When no additional input is provided, amount of data flushed is necessarily 
<= ZSTD_BLOCKSIZE_MAX.
+  Note : with no additional input provided, amount of data flushed is 
necessarily <= ZSTD_BLOCKSIZE_MAX.
  @return : 0 when a frame is completely decoded and fully flushed,
         or an error code, which can be tested using ZSTD_isError(),
         or any other value > 0, which means there is still some decoding or 
flushing to do to complete current frame :
-                                the return value is a suggested next input 
size (a hint for better latency)
-                                that will never load more than the current 
frame.
+                                the return value is a suggested next input 
size (just a hint for better latency)
+                                that will never request more than the 
remaining frame size.
  
 <BR></pre>
 
@@ -340,32 +353,477 @@ size_t ZSTD_decompressStream(ZSTD_DStream* zds, ZSTD_o
 <pre><b>size_t ZSTD_DStreamOutSize(void);   </b>/*!< recommended size for 
output buffer. Guarantee to successfully flush at least one complete block in 
all circumstances. */<b>
 </b></pre><BR>
 <a name="Chapter11"></a><h2>ADVANCED AND EXPERIMENTAL FUNCTIONS</h2><pre>
- The definitions in this section are considered experimental.
- They should never be used with a dynamic library, as prototypes may change in 
the future.
+ The definitions in the following section are considered experimental.
  They are provided for advanced scenarios.
+ They should never be used with a dynamic library, as prototypes may change in 
the future.
  Use them only in association with static linking.
  
 <BR></pre>
 
+<a name="Chapter12"></a><h2>Candidate API for promotion to stable 
status</h2><pre>
+ The following symbols and constants form the "staging area" :
+ they are considered to join "stable API" by v1.4.0.
+ The proposal is written so that it can be made stable "as is",
+ though it's still possible to suggest improvements.
+ Staging is in fact last chance for changes,
+ the API is locked once reaching "stable" status.
+ 
+<BR></pre>
+
 <pre><b>int ZSTD_minCLevel(void);  </b>/*!< minimum negative compression level 
allowed */<b>
 </b></pre><BR>
-<pre><b>typedef enum { ZSTD_fast=1, ZSTD_dfast, ZSTD_greedy, ZSTD_lazy, 
ZSTD_lazy2,
-               ZSTD_btlazy2, ZSTD_btopt, ZSTD_btultra } ZSTD_strategy;   
</b>/* from faster to stronger */<b>
+<pre><b>size_t ZSTD_findFrameCompressedSize(const void* src, size_t srcSize);
+</b><p> `src` should point to the start of a ZSTD frame or skippable frame.
+ `srcSize` must be >= first frame size
+ @return : the compressed size of the first frame starting at `src`,
+           suitable to pass as `srcSize` to `ZSTD_decompress` or similar,
+        or an error code if input is invalid 
+</p></pre><BR>
+
+<pre><b>size_t ZSTD_sizeof_CCtx(const ZSTD_CCtx* cctx);
+size_t ZSTD_sizeof_DCtx(const ZSTD_DCtx* dctx);
+size_t ZSTD_sizeof_CStream(const ZSTD_CStream* zcs);
+size_t ZSTD_sizeof_DStream(const ZSTD_DStream* zds);
+size_t ZSTD_sizeof_CDict(const ZSTD_CDict* cdict);
+size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict);
+</b><p>  These functions give the _current_ memory usage of selected object.
+  Note that object memory usage can evolve (increase or decrease) over time. 
+</p></pre><BR>
+
+<a name="Chapter13"></a><h2>Advanced compression API</h2><pre></pre>
+
+<pre><b>typedef enum { ZSTD_fast=1,
+               ZSTD_dfast=2,
+               ZSTD_greedy=3,
+               ZSTD_lazy=4,
+               ZSTD_lazy2=5,
+               ZSTD_btlazy2=6,
+               ZSTD_btopt=7,
+               ZSTD_btultra=8,
+               ZSTD_btultra2=9
+               </b>/* note : new strategies _might_ be added in the future.<b>
+                         Only the order (from fast to strong) is guaranteed */
+} ZSTD_strategy;
 </b></pre><BR>
+<pre><b>typedef enum {
+
+    </b>/* compression parameters */<b>
+    ZSTD_c_compressionLevel=100, </b>/* Update all compression parameters 
according to pre-defined cLevel table<b>
+                              * Default level is ZSTD_CLEVEL_DEFAULT==3.
+                              * Special: value 0 means default, which is 
controlled by ZSTD_CLEVEL_DEFAULT.
+                              * Note 1 : it's possible to pass a negative 
compression level.
+                              * Note 2 : setting a level sets all default 
values of other compression parameters */
+    ZSTD_c_windowLog=101,    </b>/* Maximum allowed back-reference distance, 
expressed as power of 2.<b>
+                              * Must be clamped between ZSTD_WINDOWLOG_MIN and 
ZSTD_WINDOWLOG_MAX.
+                              * Special: value 0 means "use default windowLog".
+                              * Note: Using a windowLog greater than 
ZSTD_WINDOWLOG_LIMIT_DEFAULT
+                              *       requires explicitly allowing such window 
size at decompression stage if using streaming. */
+    ZSTD_c_hashLog=102,      </b>/* Size of the initial probe table, as a 
power of 2.<b>
+                              * Resulting memory usage is (1 << (hashLog+2)).
+                              * Must be clamped between ZSTD_HASHLOG_MIN and 
ZSTD_HASHLOG_MAX.
+                              * Larger tables improve compression ratio of 
strategies <= dFast,
+                              * and improve speed of strategies > dFast.
+                              * Special: value 0 means "use default hashLog". 
*/
+    ZSTD_c_chainLog=103,     </b>/* Size of the multi-probe search table, as a 
power of 2.<b>
+                              * Resulting memory usage is (1 << (chainLog+2)).
+                              * Must be clamped between ZSTD_CHAINLOG_MIN and 
ZSTD_CHAINLOG_MAX.
+                              * Larger tables result in better and slower 
compression.
+                              * This parameter is useless when using "fast" 
strategy.
+                              * It's still useful when using "dfast" strategy,
+                              * in which case it defines a secondary probe 
table.
+                              * Special: value 0 means "use default chainLog". 
*/
+    ZSTD_c_searchLog=104,    </b>/* Number of search attempts, as a power of 
2.<b>
+                              * More attempts result in better and slower 
compression.
+                              * This parameter is useless when using "fast" 
and "dFast" strategies.
+                              * Special: value 0 means "use default 
searchLog". */
+    ZSTD_c_minMatch=105,     </b>/* Minimum size of searched matches.<b>
+                              * Note that Zstandard can still find matches of 
smaller size,
+                              * it just tweaks its search algorithm to look 
for this size and larger.
+                              * Larger values increase compression and 
decompression speed, but decrease ratio.
+                              * Must be clamped between ZSTD_MINMATCH_MIN and 
ZSTD_MINMATCH_MAX.
+                              * Note that currently, for all strategies < 
btopt, effective minimum is 4.
+                              *                    , for all strategies > 
fast, effective maximum is 6.
+                              * Special: value 0 means "use default 
minMatchLength". */
+    ZSTD_c_targetLength=106, </b>/* Impact of this field depends on 
strategy.<b>
+                              * For strategies btopt, btultra & btultra2:
+                              *     Length of Match considered "good enough" 
to stop search.
+                              *     Larger values make compression stronger, 
and slower.
+                              * For strategy fast:
+                              *     Distance between match sampling.
+                              *     Larger values make compression faster, and 
weaker.
+                              * Special: value 0 means "use default 
targetLength". */
+    ZSTD_c_strategy=107,     </b>/* See ZSTD_strategy enum definition.<b>
+                              * The higher the value of selected strategy, the 
more complex it is,
+                              * resulting in stronger and slower compression.
+                              * Special: value 0 means "use default strategy". 
*/
+
+    </b>/* LDM mode parameters */<b>
+    ZSTD_c_enableLongDistanceMatching=160, </b>/* Enable long distance 
matching.<b>
+                                     * This parameter is designed to improve 
compression ratio
+                                     * for large inputs, by finding large 
matches at long distance.
+                                     * It increases memory usage and window 
size.
+                                     * Note: enabling this parameter increases 
default ZSTD_c_windowLog to 128 MB
+                                     * except when expressly set to a 
different value. */
+    ZSTD_c_ldmHashLog=161,   </b>/* Size of the table for long distance 
matching, as a power of 2.<b>
+                              * Larger values increase memory usage and 
compression ratio,
+                              * but decrease compression speed.
+                              * Must be clamped between ZSTD_HASHLOG_MIN and 
ZSTD_HASHLOG_MAX
+                              * default: windowlog - 7.
+                              * Special: value 0 means "automatically 
determine hashlog". */
+    ZSTD_c_ldmMinMatch=162,  </b>/* Minimum match size for long distance 
matcher.<b>
+                              * Larger/too small values usually decrease 
compression ratio.
+                              * Must be clamped between ZSTD_LDM_MINMATCH_MIN 
and ZSTD_LDM_MINMATCH_MAX.
+                              * Special: value 0 means "use default value" 
(default: 64). */
+    ZSTD_c_ldmBucketSizeLog=163, </b>/* Log size of each bucket in the LDM 
hash table for collision resolution.<b>
+                              * Larger values improve collision resolution but 
decrease compression speed.
+                              * The maximum value is 
ZSTD_LDM_BUCKETSIZELOG_MAX.
+                              * Special: value 0 means "use default value" 
(default: 3). */
+    ZSTD_c_ldmHashRateLog=164, </b>/* Frequency of inserting/looking up 
entries into the LDM hash table.<b>
+                              * Must be clamped between 0 and 
(ZSTD_WINDOWLOG_MAX - ZSTD_HASHLOG_MIN).
+                              * Default is MAX(0, (windowLog - ldmHashLog)), 
optimizing hash table usage.
+                              * Larger values improve compression speed.
+                              * Deviating far from default value will likely 
result in a compression ratio decrease.
+                              * Special: value 0 means "automatically 
determine hashRateLog". */
+
+    </b>/* frame parameters */<b>
+    ZSTD_c_contentSizeFlag=200, </b>/* Content size will be written into frame 
header _whenever known_ (default:1)<b>
+                              * Content size must be known at the beginning of 
compression.
+                              * This is automatically the case when using 
ZSTD_compress2(),
+                              * For streaming variants, content size must be 
provided with ZSTD_CCtx_setPledgedSrcSize() */
+    ZSTD_c_checksumFlag=201, </b>/* A 32-bits checksum of content is written 
at end of frame (default:0) */<b>
+    ZSTD_c_dictIDFlag=202,   </b>/* When applicable, dictionary's ID is 
written into frame header (default:1) */<b>
+
+    </b>/* multi-threading parameters */<b>
+    </b>/* These parameters are only useful if multi-threading is enabled 
(compiled with build macro ZSTD_MULTITHREAD).<b>
+     * They return an error otherwise. */
+    ZSTD_c_nbWorkers=400,    </b>/* Select how many threads will be spawned to 
compress in parallel.<b>
+                              * When nbWorkers >= 1, triggers asynchronous 
mode when used with ZSTD_compressStream*() :
+                              * ZSTD_compressStream*() consumes input and 
flush output if possible, but immediately gives back control to caller,
+                              * while compression work is performed in 
parallel, within worker threads.
+                              * (note : a strong exception to this rule is 
when first invocation of ZSTD_compressStream2() sets ZSTD_e_end :
+                              *  in which case, ZSTD_compressStream2() 
delegates to ZSTD_compress2(), which is always a blocking call).
+                              * More workers improve speed, but also increase 
memory usage.
+                              * Default value is `0`, aka "single-threaded 
mode" : no worker is spawned, compression is performed inside Caller's thread, 
all invocations are blocking */
+    ZSTD_c_jobSize=401,      </b>/* Size of a compression job. This value is 
enforced only when nbWorkers >= 1.<b>
+                              * Each compression job is completed in parallel, 
so this value can indirectly impact the nb of active threads.
+                              * 0 means default, which is dynamically 
determined based on compression parameters.
+                              * Job size must be a minimum of overlap size, or 
1 MB, whichever is largest.
+                              * The minimum size is automatically and 
transparently enforced */
+    ZSTD_c_overlapLog=402,   </b>/* Control the overlap size, as a fraction of 
window size.<b>
+                              * The overlap size is an amount of data reloaded 
from previous job at the beginning of a new job.
+                              * It helps preserve compression ratio, while 
each job is compressed in parallel.
+                              * This value is enforced only when nbWorkers >= 
1.
+                              * Larger values increase compression ratio, but 
decrease speed.
+                              * Possible values range from 0 to 9 :
+                              * - 0 means "default" : value will be determined 
by the library, depending on strategy
+                              * - 1 means "no overlap"
+                              * - 9 means "full overlap", using a full window 
size.
+                              * Each intermediate rank increases/decreases 
load size by a factor 2 :
+                              * 9: full window;  8: w/2;  7: w/4;  6: w/8;  
5:w/16;  4: w/32;  3:w/64;  2:w/128;  1:no overlap;  0:default
+                              * default value varies between 6 and 9, 
depending on strategy */
+
+    </b>/* note : additional experimental parameters are also available<b>
+     * within the experimental section of the API.
+     * At the time of this writing, they include :
+     * ZSTD_c_rsyncable
+     * ZSTD_c_format
+     * ZSTD_c_forceMaxWindow
+     * ZSTD_c_forceAttachDict
+     * Because they are not stable, it's necessary to define 
ZSTD_STATIC_LINKING_ONLY to access them.
+     * note : never ever use experimentalParam? names directly;
+     *        also, the enums values themselves are unstable and can still 
change.
+     */
+     ZSTD_c_experimentalParam1=500,
+     ZSTD_c_experimentalParam2=10,
+     ZSTD_c_experimentalParam3=1000,
+     ZSTD_c_experimentalParam4=1001
+} ZSTD_cParameter;
+</b></pre><BR>
 <pre><b>typedef struct {
-    unsigned windowLog;      </b>/**< largest match distance : larger == more 
compression, more memory needed during decompression */<b>
-    unsigned chainLog;       </b>/**< fully searched segment : larger == more 
compression, slower, more memory (useless for fast) */<b>
-    unsigned hashLog;        </b>/**< dispatch table : larger == faster, more 
memory */<b>
-    unsigned searchLog;      </b>/**< nb of searches : larger == more 
compression, slower */<b>
-    unsigned searchLength;   </b>/**< match length searched : larger == faster 
decompression, sometimes less compression */<b>
-    unsigned targetLength;   </b>/**< acceptable match size for optimal parser 
(only) : larger == more compression, slower */<b>
-    ZSTD_strategy strategy;
+    size_t error;
+    int lowerBound;
+    int upperBound;
+} ZSTD_bounds;
+</b></pre><BR>
+<pre><b>ZSTD_bounds ZSTD_cParam_getBounds(ZSTD_cParameter cParam);
+</b><p>  All parameters must belong to an interval with lower and upper bounds,
+  otherwise they will either trigger an error or be automatically clamped.
+ @return : a structure, ZSTD_bounds, which contains
+         - an error status field, which must be tested using ZSTD_isError()
+         - lower and upper bounds, both inclusive
+ 
+</p></pre><BR>
+
+<pre><b>size_t ZSTD_CCtx_setParameter(ZSTD_CCtx* cctx, ZSTD_cParameter param, 
int value);
+</b><p>  Set one compression parameter, selected by enum ZSTD_cParameter.
+  All parameters have valid bounds. Bounds can be queried using 
ZSTD_cParam_getBounds().
+  Providing a value beyond bound will either clamp it, or trigger an error 
(depending on parameter).
+  Setting a parameter is generally only possible during frame initialization 
(before starting compression).
+  Exception : when using multi-threading mode (nbWorkers >= 1),
+              the following parameters can be updated _during_ compression 
(within same frame):
+              => compressionLevel, hashLog, chainLog, searchLog, minMatch, 
targetLength and strategy.
+              new parameters will be active for next job only (after a 
flush()).
+ @return : an error code (which can be tested using ZSTD_isError()).
+ 
+</p></pre><BR>
+
+<pre><b>size_t ZSTD_CCtx_setPledgedSrcSize(ZSTD_CCtx* cctx, unsigned long long 
pledgedSrcSize);
+</b><p>  Total input data size to be compressed as a single frame.
+  Value will be written in frame header, unless if explicitly forbidden using 
ZSTD_c_contentSizeFlag.
+  This value will also be controlled at end of frame, and trigger an error if 
not respected.
+ @result : 0, or an error code (which can be tested with ZSTD_isError()).
+  Note 1 : pledgedSrcSize==0 actually means zero, aka an empty frame.
+           In order to mean "unknown content size", pass constant 
ZSTD_CONTENTSIZE_UNKNOWN.
+           ZSTD_CONTENTSIZE_UNKNOWN is default value for any new frame.
+  Note 2 : pledgedSrcSize is only valid once, for the next frame.
+           It's discarded at the end of the frame, and replaced by 
ZSTD_CONTENTSIZE_UNKNOWN.
+  Note 3 : Whenever all input data is provided and consumed in a single round,
+           for example with ZSTD_compress2(),
+           or invoking immediately ZSTD_compressStream2(,,,ZSTD_e_end),
+           this value is automatically overriden by srcSize instead.
+ 
+</p></pre><BR>
+
+<pre><b>size_t ZSTD_CCtx_loadDictionary(ZSTD_CCtx* cctx, const void* dict, 
size_t dictSize);
+</b><p>  Create an internal CDict from `dict` buffer.
+  Decompression will have to use same dictionary.
+ @result : 0, or an error code (which can be tested with ZSTD_isError()).
+  Special: Loading a NULL (or 0-size) dictionary invalidates previous 
dictionary,
+           meaning "return to no-dictionary mode".
+  Note 1 : Dictionary is sticky, it will be used for all future compressed 
frames.
+           To return to "no-dictionary" situation, load a NULL dictionary (or 
reset parameters).
+  Note 2 : Loading a dictionary involves building tables.
+           It's also a CPU consuming operation, with non-negligible impact on 
latency.
+           Tables are dependent on compression parameters, and for this reason,
+           compression parameters can no longer be changed after loading a 
dictionary.
+  Note 3 :`dict` content will be copied internally.
+           Use experimental ZSTD_CCtx_loadDictionary_byReference() to 
reference content instead.
+           In such a case, dictionary buffer must outlive its users.
+  Note 4 : Use ZSTD_CCtx_loadDictionary_advanced()
+           to precisely select how dictionary content must be interpreted. 
+</p></pre><BR>
+
+<pre><b>size_t ZSTD_CCtx_refCDict(ZSTD_CCtx* cctx, const ZSTD_CDict* cdict);

*** DIFF OUTPUT TRUNCATED AT 1000 LINES ***
_______________________________________________
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"

Reply via email to