> Are there old Hive/Hadoop version combos we should just stop supporting?
Dropping support for Java 8 means dropping support for Hive lower than 2.0(exclusive)[1]. IsolatedClientLoader is aimed to allow using different Hive jars to communicate with different versions of HMS. AFAIK, the current built-in Hive 2.3.9 client works well on communicating with the Hive Metastore server through 2.1 to 3.1 (maybe 2.0 too, not sure). This brings a new question, does the IsolatedClientLoader required then? I think we should drop IsolatedClientLoader because 1. As explained above, we can use the built-in Hive 2.3.9 client to communicate with HMS 2.1+ 2. Since SPARK-42539[2], the default Hive 2.3.9 client does not use IsolatedClientLoader, and as explained in SPARK-42539, IsolatedClientLoader causes some inconsistent behaviors. 3. It blocks Guava upgrading. HIVE-27560[3] aim to make Hive 2.3.10(unreleased) compatible with all Guava 14+ versions, but unfortunately, Guava is marked as `isSharedClass`[3] in IsolatedClientLoader, so technically, if we want to upgrade Guava we need to make all supported Hive versions(through 2.1.x to 3.1.x) to support high version of Guava, I think it's impossible. [1] sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientVersions.scala [2] https://issues.apache.org/jira/browse/SPARK-42539 [3] https://issues.apache.org/jira/browse/HIVE-27560 [4] https://github.com/apache/spark/pull/33989#issuecomment-926277286 Thanks, Cheng Pan > On Aug 8, 2023, at 05:47, Sean Owen <sro...@gmail.com> wrote: > > Are there old Hive/Hadoop version combos we should just stop supporting? --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
