taowenjun opened a new issue, #13087:
URL: https://github.com/apache/seatunnel/issues/13087
### Search before asking
- [x] I searched existing feature requests and found no similar feature
requirement.
### Description
Hive sink currently requires compression to be configured explicitly through
`compress_codec`, even when an existing Hive ORC table defines compression
using its `orc.compress` table property.
For example:
```sql
CREATE TABLE example_orc_table (
id BIGINT,
name STRING
)
STORED AS ORC
TBLPROPERTIES ('orc.compress' = 'SNAPPY');
```
Writing to this table without setting `compress_codec` currently uses the
default NONE rather than inheriting SNAPPY. This duplicates configuration
between Hive table metadata and SeaTunnel jobs and can cause the output
compression to differ from the table setting.
Add an opt-in Hive sink option, `use_table_compression`, defaulting to
`false` for backward compatibility. When enabled for an existing ORC table,
compression is selected in this order:
1. Explicit `compress_codec`, including `NONE`.
2. The table's `orc.compress` property.
3. The existing default if the property is missing or blank.
Table property values should be trimmed and normalized case-insensitively.
Unsupported inherited codecs should fail with a descriptive error instead of
silently producing uncompressed files. Non-ORC formats and automatic table
creation are outside this request.
### Usage Scenario
Users writing to existing Hive ORC tables want SeaTunnel to follow the
compression configured in Hive metastore, while retaining the ability to
override it for individual jobs. Existing jobs should preserve their behavior
unless the feature is explicitly enabled.
```hocon
sink {
Hive {
table_name = "test_db.existing_orc_table"
metastore_uri = "thrift://localhost:9083"
use_table_compression = true
}
}
```
### Related issues
Implementation PR: #13085.
The ORC/ZSTD bug is tracked separately in #13086, with its fix in #13084.
This feature does not upgrade ORC dependencies. On current dev, inherited ZSTD
is explicitly rejected because its ORC mapping is NONE. Once the separate
ORC/ZSTD fix is merged, the same inheritance logic can accept ZSTD.
### Are you willing to submit a PR?
- [x] Yes, I am willing to submit a PR. Implementation is available in
#13085, with builds and 31 focused unit tests passing independently on JDK 11
and JDK 17. Live Hive/HDFS integration tests were not run for the contribution
branch.
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
