jlalwani-amazon opened a new pull request, #37: URL: https://github.com/apache/flink-connector-hive/pull/37
# [FLINK-35515][hive] Add Hive 4 support to the Flink Hive connector ## What is the purpose of the change Add Apache Hive 4.x compatibility to the Flink Hive connector. Hive 4 is a major cleanup release with significant API changes (removed classes, changed return types, relocated packages, added required parameters). This PR introduces a dedicated `HiveShimV400` module and extends the existing `HiveShim` abstraction to handle all version-specific differences. *JIRA: [FLINK-35515](https://issues.apache.org/jira/browse/FLINK-35515)* ## Brief change log ### Commit 1: Add HiveConfVars for Hive 4 ConfVars enum renames Hive 4 renamed all `HiveConf.ConfVars` enum constants to snake_case (HIVE-27925, e.g. `METASTOREURIS` → `METASTORE_URIS`). Created `HiveConfVars.java` that resolves the correct enum at class-load time via `valueOf()` with fallback. 59 constants across 35 files. ### Commit 2: Add Hive 4 shim module New `flink-connector-hive-4.0.0` Maven module compiled against Hive 4 jars. `HiveShimV400` extends `HiveShimV313` and overrides methods for Hive 4 API changes. `HiveShimLoader` loads it by reflection to avoid cyclic Maven dependencies. Supports Hive 4.0.0, 4.0.1, 4.1.0, 4.2.0. ### Commit 3: Add Hive 4 source compatibility fixes 21 files changed for Hive 4 API compatibility: - `HiveFileFormatUtils.getRecordWriter()` removed → `HiveShimV400.getHiveRecordWriter()` uses `getHiveRecordWriter(JobConf, TableDesc, Class, FileSinkDesc, Path, Reporter)` - `StructTypeInfo.getAllStructFieldNames/TypeInfos()` return type `ArrayList` → `List` → `HiveShim.getStructFieldNames/TypeInfos()` - `SerDeUtils.initializeSerDe()` removed → `HiveShimV400.initializeSerDe()` calls `AbstractSerDe.initialize(conf, tableProps, partProps)` - `IMetaStoreClient.getTableColumnStatistics/getPartitionColumnStatistics` added engine parameter - `PrincipalDesc` relocated from `ql.plan` to `ql.ddl.privilege` → `HiveShim.createPrincipalDesc()` returns `Object` - `PreOrderWalker` changed to require `SemanticDispatcher` → `HiveShim.walkExpressionTree()` - `HiveMetaStoreUtils.getDeserializer/getFieldsFromDeserializer` signature changes - `RowSchema.getSignature()` return type `ArrayList` → `List` ### Commit 4: Add unit test support for Hive 4 `hive4` Maven profile with test dependency overrides: protobuf 3.3.0 (hive-exec bundles it unshaded), DataNucleus 5.x, Hadoop 3.3.6, antlr-runtime 3.5.2, `--add-opens` JVM flags. ### Commit 5: Add integration test support and fixes for Hive 4 - `HiveShim.getGenericWindowingEvaluator()` for Hive 4's 5-arg `FunctionRegistry.getGenericWindowingEvaluator` (added `respectNulls` param) - `HiveShimV400.loadTable()` for Hive 4's 11-arg `Hive.loadTable()` (added `isInsertOverwrite` param) - `HiveServer2Endpoint` includes `hive.server2.thrift.resultset.default.fetch.size` in OpenSession response (Hive 4 JDBC client requires it) - `HiveRunnerShimLoader` extended with Hive 4 version cases - Test expectation adjustments for Hive 4 behavioral changes - `HiveShimV400Test` unit tests ## Why a separate Maven module? Previous shims (V310–V313) compile against Hive 2.3 because API changes between 2.3 and 3.1 were minor. Hive 4 has far more breaking changes — removed classes, changed return types, relocated packages. Without a separate module, every API call requires reflection. The dedicated module compiles `HiveShimV400` against Hive 4 jars so all calls are direct. ## Verifying this change ### Build ```bash # Hive 2.3 (default) mvn clean install -DskipTests # Hive 4 mvn install -pl flink-connector-hive-4.0.0 -am -DskipTests mvn compile test-compile -f flink-connector-hive/pom.xml -Phive4 ``` ### Unit tests ```bash # Hive 2.3: 400 tests, 0 failures mvn test -pl flink-connector-hive -am -Dsurefire.excludes='**/*ITCase*' # Hive 4: 400 tests, 0 failures mvn install -pl flink-connector-hive-4.0.0 -am -DskipTests mvn test -f flink-connector-hive/pom.xml -Phive4 -Dsurefire.excludes='**/*ITCase*' ``` ### Integration tests ```bash # Hive 4 integration tests mvn install -pl flink-connector-hive-4.0.0 -am -DskipTests mvn test -f flink-connector-hive/pom.xml -Phive4 \ -Dtest='*ITCase,!HiveServer2DelegationTokenProviderITCase' \ -DfailIfNoTests=false ``` ## Known limitations 1. **`flink-orc` doesn't support Hive 4 yet.** `OrcShim.createShim()` in `flink-formats/flink-orc` (upstream `apache/flink` repo) throws `UnsupportedOperationException` for Hive 4. ORC batch writes work (use `OrcBulkWriterFactory` directly), but ORC reads via `HiveInputFormat` fail. Fix: add `hiveVersion.startsWith("4.")` mapping to `OrcShimV230` — needs a separate PR to `apache/flink`. 2. **Hive 4 removed numeric-to-timestamp casting.** `GenericUDFTimestamp` rejects numeric input entirely (not just a config change). Tests relying on this behavior are skipped on Hive 4. 3. **Hive 4 `INSERT OVERWRITE DIRECTORY` ignores `COLLECTION ITEMS TERMINATED BY`.** Uses default `\u0002` delimiter regardless. Test expectations adjusted. ## Does this pull request potentially affect one of the following parts? - Dependencies (does it add or upgrade a dependency): **yes** (new `flink-connector-hive-4.0.0` module with Hive 4 compile dependency; `hive4` test profile adds protobuf, DataNucleus, Hadoop overrides) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: **no** - The serializers: **no** - The runtime per-record code paths (performance sensitive): **no** - Anything that affects deployment or recovery: **no** ## Documentation - [ ] Does this pull request introduce a new feature? **yes** — Hive 4 support - [ ] If yes, how is the feature documented? **not yet** — docs update will follow -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
