rmannibucau opened a new pull request, #9229:
URL: https://github.com/apache/hop/pull/9229
# Make commons-compress the compression registry and add the missing formats
Closes the problem raised in discussion #9191: Hop only shipped None, GZip,
Zip, Snappy and Hadoop-snappy out of the box, so compressing a file with zstd
required writing a custom plugin. This PR brings the common formats into the
default distribution by making Apache Commons Compress the single
first-class compression registry.
## What changed
**14 commons-compress formats are now built-in** - zstd, gzip, bzip2, xz,
lzma, snappy-framed, snappy-raw, lz4-framed, lz4-block, deflate, deflate64,
z, pack200, brotli. Each is a metadata-only `@CompressionPlugin` class in
`org.apache.hop.core.compress.format`, discovered through the existing
plugin machinery (Jandex index) - no manual registration list, no XML.
**Hop's `none` and `zip` codecs join the same registry** through a
`CompressorStreamProvider` SPI implementation
(`org.apache.hop.core.compress.spi.HopCompressorStreamProvider`), preserving
the historical zip entry semantics (`addEntry`/`nextEntry`, folder stripping,
extension appending - verified by tests).
**`HopCompressionRegistry`** decorates the commons-compress singleton so the
Hop codecs are always available regardless of class loader (commons-compress
discovers SPI providers via the system class loader, which is unreliable in
embedded contexts). Extension codecs can be registered/unregistered
dynamically and are de-duplicated by registry key, so a plugin shipping both
an annotation class and a `META-INF/services` file is registered exactly
once.
**All format metadata moved into the `@CompressionPlugin` annotation**
(description, default extension, input/output support). The legacy
`ICompressionProvider` / `CompressionProviderFactory` API is kept and
deprecated, and its methods now delegate to the registry.
**Better dialogs** - Text File Output/Input list only the formats that
support the selected direction, keep "None" as the first entry, preserve
stored values that are absent from the filtered list, and Text File Input now
detects the format from the magic bytes before falling back to the file
extension.
## Compatibility
- Existing pipelines keep working: "GZip", "Zip", "None" resolve to the
commons-compress backed providers; xerial "Snappy" and "Hadoop-snappy"
remain registered for wire compatibility (their format differs from the
commons-compress snappy codecs).
- The legacy classes are deprecated but not removed.
## Dependencies
- `zstd-jni` bumped 1.5.7-23 -> 1.5.7-25, `xz` 1.12 and `brotli-dec` 0.1.2
added to hop-lib dependency management.
- `org.lz4:lz4-java` is deliberately **not** added: commons-compress 1.28.0
implements LZ4 in pure Java, and lz4-java has unfixed CVEs
(CVE-2025-12183, CVE-2025-66566).
## Tests
- New `CommonsCompressFormatsTest`, `HopCompressionRegistryTest`,
`SnowflakeBulkLoaderGzipInteropTest` plus extended
`CompressionProviderFactoryTest`.
- Coverage: registry merge and dedupe, annotation-only extension discovery,
round-trips and magic bytes for every built-in format, zip entry contract,
legacy name resolution, input-only format rejection, magic-byte detection.
- `mvn -pl engine test`: 1285 tests green; `textfile` and `snowflake`
plugin suites green; spotless applied.
## Docs
New "Compression formats" page in the dev manual
(`compression-formats.adoc`) explaining the architecture and how to add a
new format.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
