pjfanning opened a new pull request, #3641:
URL: https://github.com/apache/pekko/pull/3641
### Motivation
The Jackson serializers (`serialization-jackson` and
`serialization-jackson3`) can compress payloads with gzip or lz4. zstd
compresses nearly as well as gzip while being much faster, so it is a useful
third option, especially for `jackson-json`, which compresses large payloads by
default.
### Modification
- **New setting:** `compression.algorithm = zstd`, plus
`compression.zstd-level` (default 3, zstd's default level). The level is
validated when the serializer is created.
- **Optional dependency:** zstd-jni is optional, as in #3639:
`<optional>true</optional>` in the POM and `resolution:=optional` in the OSGi
`Import-Package`.
- All zstd-jni calls are in a separate internal `ZstdCompression` object.
- Configuring `algorithm = zstd` without zstd-jni, or receiving a zstd
payload without it, fails with an `IllegalArgumentException` naming the missing
dependency, not a `LinkageError`.
- **Detection:** zstd payloads are recognised on deserialization by the
standard zstd frame magic number (`28 B5 2F FD`), whatever `algorithm` is
configured, like gzip and lz4. No custom header is added.
- **Decompression bombs:** `compression.max-decompressed-size` applies to
zstd as well.
- A frame that declares a decompressed size larger than the limit is
rejected before anything is allocated.
- Otherwise the declared size is the capacity of the output, so a frame
that produces more than it declares fails instead of growing past the checked
size.
- Frames that don't declare a size (e.g. written by a streaming
compressor) are decompressed in chunks and rejected as soon as the output
passes the limit, like gzip.
- **Buffer sizing:** the `ByteArrayOutputStream`s used for gzip compression,
gunzip and the zstd chunked path now start at a size based on the input.
- Decompression: 4x the compressed size, at least 4 KiB and at most 16
MiB, and never more than `max-decompressed-size` allows. Before, they started
at Java's default of 32 bytes.
- gzip compression: a quarter of the input size, at least 4 KiB. Before,
it was a fixed 4 KiB.
- **Docs:** `serialization-jackson.md` covers zstd, adds a dependency
snippet, and explains how to switch a running cluster to zstd: first roll out
zstd-jni with the old algorithm, then change `algorithm` in a second rolling
update.
zstd dictionaries are not supported. They would mainly help small messages,
but they need every node to share the same dictionaries, which is better
handled in a follow-up.
### Result
Jackson payloads can be compressed with zstd, with the same protection
against oversized decompression as gzip and lz4.
### Tests
- `sbt "serialization-jackson/test" "serialization-jackson3/test"`: passed
(161 and 159 tests). New tests:
- compressing large payloads and not compressing small ones;
- `zstd-level` is used, and an invalid level is rejected;
- zstd payloads are read whatever the configured algorithm;
- rejection beyond `max-decompressed-size`, both for declared sizes and
for frames without one;
- rejection of a payload that produces more than its declared size;
- `-1` meaning no limit;
- buffer sizing bounds;
- `JacksonWithoutZstdSpec`: gzip works, and zstd use fails clearly, in a
class loader without zstd-jni.
- `sbt "serialization-jackson/mimaReportBinaryIssues"`: passed.
serialization-jackson3 has no MiMa setup, since it hasn't been released yet.
- `sbt "serialization-jackson/makePom" "serialization-jackson/packageBin"`:
zstd-jni is optional in the POM and in the OSGi `Import-Package`.
### References
Refs #3639 (zstd for Pekko Streams). Both PRs add `zstdJniVersion`/`zstdJni`
to `project/Dependencies.scala` and `zstd_jni_version` to
`project/Paradox.scala`, so whichever merges second needs a small rebase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
