taowenjun opened a new pull request, #13085:
URL: https://github.com/apache/seatunnel/pull/13085
### Purpose of this pull request
This PR adds an opt-in option for Hive sink to inherit compression from an
existing ORC table's `orc.compress` property, avoiding duplicated compression
settings in job configurations.
### Does this PR introduce _any_ user-facing change?
Yes. Add `use_table_compression`, a boolean option with a default value of
`false`.
When enabled for an existing ORC table, compression is selected in the
following order:
1. Explicitly configured `compress_codec`, including `NONE`.
2. The table's `orc.compress` property.
3. The existing default if the table property is missing or blank.
Table property values are trimmed and normalized case-insensitively.
Unsupported inherited codecs fail with a descriptive error instead of silently
producing uncompressed files.
Example:
```hocon
sink {
Hive {
table_name = "test_db.existing_orc_table"
metastore_uri = "thrift://localhost:9083"
use_table_compression = true
}
}
```
Existing jobs retain their behavior because the option defaults to `false`.
Non-ORC tables and automatic table creation are unaffected.
This PR is independent of #13084. On current `dev`, ZSTD maps to ORC NONE,
so inheriting `orc.compress=ZSTD` fails explicitly. After #13084 fixes the ORC
dependency and compression mapping, this inheritance logic will accept ZSTD.
### How was this patch tested?
Extended existing unit tests:
- `HiveSinkConfigTest`: inheritance, normalization, explicit overrides,
missing/blank properties, non-ORC isolation, and unsupported codecs.
- `HiveSinkOptionsTest`: default value and configuration parsing.
- `HiveSinkFactoryTest`: option registration.
Verified with JDK 11.0.33 and JDK 17.0.21:
```bash
mvn -B -ntp \
-pl seatunnel-connectors-v2/connector-hive -am \
-Dtest=HiveSinkConfigTest,HiveSinkOptionsTest,HiveSinkFactoryTest \
-Dsurefire.failIfNoSpecifiedTests=false \
clean verify
```
Both builds passed across 20 modules, with 31 tests passing and no failures,
errors, or skips.
Configuration tests mock Hive metastore metadata and instantiate the real
Hive sink to check the generated file sink configuration. Live Hive/HDFS
integration tests were not run.
### Check list
- No new dependencies or binary packages are introduced.
- No incompatible default behavior changes are introduced.
- No new connector is added; connector registration changes are not required.
- Documentation and E2E tests are not included in this patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
