Hi everyone, We'd like to propose adding Apache Hive 4.x support to the externalized Hive connector (https://github.com/apache/flink-connector-hive).
Motivation ---------- Apache Hive 4.0.0 was released in March 2024, with 4.1.0 following later that year. Hive 4 introduces significant changes including Iceberg integration, HMS Thrift-over-HTTP transport, JWT authentication, Calcite 1.25, and standalone metastore improvements. The Flink Hive connector currently supports only Hive 2.3.x and 3.1.x. There is an existing JIRA for this (FLINK-35515) but no prior discussion on the mailing list and no active work. We are aware that the Hive connector externalization is still in progress — FLINK-30064 remains open, FLINK-37094 (adapting to Flink 2.0 API changes) is unassigned, and the external repo has not yet published a release. We are willing to help with these items as part of this effort. Scope ----- We would target Flink 2.x only. Since the Hive connector has already been removed from core Flink's master branch (FLINK-33786), and the external repo needs Flink 2.0 API adaptation work regardless (FLINK-37094), it makes sense to do Hive 4 support on top of a Flink 2.x-compatible connector rather than backporting to 1.20. The work would include: 1. Help resolve FLINK-37094 (adapt the external connector to Flink 2.0 API changes) as a prerequisite 2. Add a HiveShimV400 following the existing shim pattern in org.apache.flink.table.catalog.hive.client 3. Add a flink-sql-connector-hive-4.0.0 bundled JAR module 4. Update HiveShimLoader to recognize Hive 4.x versions 5. Handle known breaking changes: - Deprecated HMS Thrift APIs (HIVE-26537) - Thrift-over-HTTP transport support (HIVE-21456) - Calcite version alignment (Hive 4 uses 1.25, Flink uses a shaded version) - Dependency updates (Hadoop 3.3.6 baseline) 6. Update documentation and tests Questions for the community --------------------------- - Would this require a FLIP, or is the existing FLINK-35515 JIRA sufficient given that this follows the established shim pattern? - Are there any concerns about the Calcite version conflict between Hive 4 and Flink's shaded Calcite? - Is there a committer willing to shepherd this work through review? We'd appreciate feedback on the approach. Thanks, Jayesh Lalwani
