Hi all, Following up on this thread. I've done some preliminary analysis and wanted to share findings, and also offer to take on FLINK-37094.
Flink 2.0 compatibility (FLINK-37094) -------------------------------------- I compiled the external connector against Flink 2.0.0 and cataloged all the breakages. There are ~100 compilation errors across 13 files, falling into these categories: - Legacy Table API removed (TableSource, TableSink, TableSourceFactory, etc.) - 4 files - UniqueConstraint removed from table.api.constraints - 7 files including the HiveShim interface - StreamingFileSink/BucketsBuilder removed - HiveTableSink.java - ManagedTableFactory/ManagedTableListener removed - HiveCatalog, HiveDynamicTableFactory - CatalogLock/RequireCatalogLock removed - HiveCatalog, HiveCatalogLock - flink-hadoop-compatibility artifact dropped the _2.12 Scala suffix I'd like to take on FLINK-37094 and submit a PR for these fixes. This would be a prerequisite for any further work on the external connector, including Hive 4 support. Hive 4 compatibility --------------------- I also compiled against hive-exec:4.0.0 and the results are encouraging - only 5 compilation errors across 8 files, all in the SQL parser/planner layer (class relocations and removals in Hive 4). Notably, the deprecated HMS Thrift APIs (HIVE-26537) that we listed as a known breaking change in the original proposal do not actually affect this connector - the IMetaStoreClient interface is unchanged in Hive 4. Next steps ---------- I'm ready to start on the FLINK-37094 fixes immediately. Would a committer be willing to review a PR for the Flink 2.0 adaptation? Once that's in, the Hive 4 changes are relatively small on top. Thanks, Jayesh On 4/8/26, 3:23 PM, "Lalwani, Jayesh" <[email protected] <mailto:[email protected]>> wrote: CAUTION: Email security compliance checks failed. Do not reply to this email, click links or open attachments unless you can confirm the sender. For more information, search for DMARC on Amazon's IT Portal at it.amazon.com Hi everyone, We'd like to propose adding Apache Hive 4.x support to the externalized Hive connector (https://github.com/apache/flink-connector-hive <https://github.com/apache/flink-connector-hive>). Motivation ---------- Apache Hive 4.0.0 was released in March 2024, with 4.1.0 following later that year. Hive 4 introduces significant changes including Iceberg integration, HMS Thrift-over-HTTP transport, JWT authentication, Calcite 1.25, and standalone metastore improvements. The Flink Hive connector currently supports only Hive 2.3.x and 3.1.x. There is an existing JIRA for this (FLINK-35515) but no prior discussion on the mailing list and no active work. We are aware that the Hive connector externalization is still in progress — FLINK-30064 remains open, FLINK-37094 (adapting to Flink 2.0 API changes) is unassigned, and the external repo has not yet published a release. We are willing to help with these items as part of this effort. Scope ----- We would target Flink 2.x only. Since the Hive connector has already been removed from core Flink's master branch (FLINK-33786), and the external repo needs Flink 2.0 API adaptation work regardless (FLINK-37094), it makes sense to do Hive 4 support on top of a Flink 2.x-compatible connector rather than backporting to 1.20. The work would include: 1. Help resolve FLINK-37094 (adapt the external connector to Flink 2.0 API changes) as a prerequisite 2. Add a HiveShimV400 following the existing shim pattern in org.apache.flink.table.catalog.hive.client 3. Add a flink-sql-connector-hive-4.0.0 bundled JAR module 4. Update HiveShimLoader to recognize Hive 4.x versions 5. Handle known breaking changes: - Deprecated HMS Thrift APIs (HIVE-26537) - Thrift-over-HTTP transport support (HIVE-21456) - Calcite version alignment (Hive 4 uses 1.25, Flink uses a shaded version) - Dependency updates (Hadoop 3.3.6 baseline) 6. Update documentation and tests Questions for the community --------------------------- - Would this require a FLIP, or is the existing FLINK-35515 JIRA sufficient given that this follows the established shim pattern? - Are there any concerns about the Calcite version conflict between Hive 4 and Flink's shaded Calcite? - Is there a committer willing to shepherd this work through review? We'd appreciate feedback on the approach. Thanks, Jayesh Lalwani
