wypoon commented on PR #13262: URL: https://github.com/apache/iceberg/pull/13262#issuecomment-2956886592
> According to changes in `hive-metastore/src/main`, I suppose the iceberg-hive-metastore.jar compiled by Hive 2.3.10 is also binary compatible with Hive 3.x and 4.x runtime jars? @pan3793 I don't believe so. According to @danielcweeks in https://github.com/apache/iceberg/pull/12721#issuecomment-2855780956, there is a binary incompatibility. We cannot use iceberg-hive-metastore.jar built against Hive 2 with a Hive 4 HMS or vice versa. We discussed this in the last community sync. As I understand it, the way forward is to build the iceberg-hive-metastore against separate versions of Hive, producing separate artifacts, and to remove the bundled classes from the runtime jars of the engines. Users of the engines can then choose which version of iceberg-hive-metastore.jar they wish to use, according to the version of HMS they use. However, removal of the bundled classes is a breaking change and won't be done until Iceberg 2.0. I know you have raised the point that Spark's IsolatedClientLoader is only used by its HiveExternalCatalog. However, here in this PoC, what we're doing is to put the Hive 4 iceberg-hive-metastore classes as well as Hive 4 classes in the Spark classpath for the tests, so Spark is loading Hive 4 metastore client classes to talk to the Iceberg `TestHiveMetastore`. It appears that for Spark 4.0, this works. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
