taowenjun opened a new pull request, #13129:
URL: https://github.com/apache/seatunnel/pull/13129
## Purpose of this change
This contribution to Apache SeaTunnel extends the opt-in Hive
table-compression inheritance introduced by #13085 to existing Parquet tables.
A table declaring `parquet.compression=SNAPPY` can otherwise receive
uncompressed output unless every job repeats the codec.
Closes #13128. This is a separate follow-up requested in #13087, not an
expansion of that ORC issue.
## Resulting behavior
- `use_table_compression` stays disabled by default. For existing Parquet
tables, explicit `compress_codec` (including `NONE`) takes precedence over
`parquet.compression`, then the existing default.
- Trim and normalize inherited values; map Parquet `UNCOMPRESSED` to `NONE`.
Missing, null and blank properties keep existing defaults.
- Reject unknown values and inherited `ZLIB`, whose Parquet mapping would
silently write uncompressed output. Explicit codec behavior and ORC behavior
are unchanged.
- Skip automatic table creation and unrelated formats. Do not change Hive
metadata.
- Document behavior and add local footer/data checks and Hive integration
scenarios for inherited SNAPPY and explicit NONE.
## Dependency and validation
**Draft: depends on #13085.** This branch currently includes that unmerged
ORC prerequisite rebased onto dev at `2eccdc412`. The Parquet delta is isolated
in commit `6543fe8c9`. After #13085 lands, rebase onto dev to remove
prerequisite commits before marking this PR ready for review. Full Hive E2E
execution is also pending.
JDK 11 and JDK 17 targeted Maven verify runs each executed 36 Hive tests
with zero failures, errors or skipped tests. Actual local Parquet files verify
footer codecs and exact three-row round trips including NULL, covering SNAPPY
inheritance, NONE override, UNCOMPRESSED alias and GZIP inheritance. These
local writer tests use the resolved Hive configuration with Avro's Parquet
writer; they do not substitute for the full production Hive sink pipeline.
Hive Docker integration scenarios are included. The Hive E2E reactor passed
Maven verify with `-DskipTests -Dskip.ui=true`, including compilation of the
new HiveIT scenarios. This is compilation/package evidence only: Docker is
unavailable locally, so integration execution is not verified. The user offered
the existing Azkaban `upgrade_validation` project, but its saved session is
expired (`{"error":"session"}`), confirmed again on request; no project was
uploaded or executed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]