taowenjun opened a new pull request, #13084:
URL: https://github.com/apache/seatunnel/pull/13084
### Purpose of this pull request
Yes, I am contributing this fix to Apache SeaTunnel.
Fix ZSTD-compressed ORC reads and writes in the file connector.
The current ORC 1.5.6 dependency does not recognize ZSTD compression,
causing reads to fail with:
```text
java.lang.IllegalArgumentException: Unknown compression
```
Additionally, `CompressFormat.ZSTD` maps to ORC `CompressionKind.NONE`, so
selecting ZSTD for ORC output silently produces uncompressed files.
This patch:
- Upgrades ORC from 1.5.6 to 1.8.7.
- Retains the `nohive` classifier and relocated storage classes to avoid
conflicts with Hive runtime dependencies.
- Maps ZSTD to ORC `CompressionKind.ZSTD`.
- Excludes transitive Hadoop client API/runtime artifacts so the connector
continues using SeaTunnel's existing Hadoop dependencies.
- Adds regression coverage to the existing `OrcWriteStrategyTest`.
### Does this PR introduce _any_ user-facing change?
Yes.
Before this patch:
- Reading ZSTD-compressed ORC files fails with `Unknown compression`.
- Writing ORC with `compress_codec = "zstd"` produces files whose footer
reports `NONE`.
After this patch:
- ZSTD-compressed ORC files can be read.
- Writing ORC with `compress_codec = "zstd"` produces files whose footer
reports `ZSTD`.
No configuration names or default values change. Existing uncompressed ORC
files remain readable.
The patch is based on dev commit `b434d2e4f46807a82b0cf4b1a6b52c1e88fd0532`.
### How was this patch tested?
Added unit tests covering:
- The ZSTD-to-ORC compression mapping.
- Real-file round trips using ZSTD, SNAPPY, NONE, ZLIB and LZ4.
- Actual ORC footer compression and row count.
- Inferred schema and exact integer, string, decimal and nanosecond
timestamp values.
- Unicode strings and null values.
The footer assertion ensures an uncompressed round trip cannot incorrectly
pass as ZSTD support.
Before the fix, a regression check against the original compression mapping
failed with:
```text
Expected ORC ZSTD, actual NONE
```
Verified independently with JDK 11.0.33 and JDK 17.0.21:
```bash
mvn -B -ntp \
-pl seatunnel-connectors-v2/connector-file/connector-file-base \
-am \
-Dtest=OrcWriteStrategyTest,OrcReadStrategyTest \
-Dsurefire.failIfNoSpecifiedTests=false \
clean verify
```
Both runs completed successfully:
- All 14 reactor modules built successfully.
- 14 tests discovered: 11 passed and 3 pre-existing tests skipped because
they are disabled on Windows.
- All newly added tests executed.
- Spotless checks passed.
Validation was performed on Windows. Docker E2E tests and live Hive jobs
were not run for this branch.
### Check list
* [ ] Dependency license/notice review.
* [ ] Documentation updates, if required.
* [ ] Incompatible-change documentation, if required.
* [ ] Connector registration and E2E updates, if required.
This patch changes an existing connector and adds no new connector or
checked-in JAR binary. Connector registration files are unchanged.
The submitted changes are limited to the dependency POM, compression mapping
and unit tests. Documentation and E2E files are not included in this patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
