pan3793 commented on PR #42493: URL: https://github.com/apache/spark/pull/42493#issuecomment-1678857243
> We should probably do something like this for 4.0, yes. @srowen yes, it's target to 4.0 > Why does it require removing the isolated classloader? Guava is marked as shared, which means that Guava is not isolated totally. (I don't understand the background, but the fact is, it's required, excluding it from shared classes breaks the test) https://github.com/apache/spark/blob/afcccb42c96cc7785a85c9463523f34d7a900a1d/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala#L205C3-L219 So technically, if we want to upgrade Guava with `IsolatedClientLoader`, we should make all Hive versions compatible with the new Guava, it's impossible. And as I comment in https://www.mail-archive.com/[email protected]/msg30708.html, I think the `IsolatedClientLoader` is not required after dropping Java 8 > Dropping support for Java 8 means dropping support for Hive lower than 2.0(exclusive)[1]. > > IsolatedClientLoader aims to allow using different Hive jars to communicate with different versions of HMS. AFAIK, the current built-in Hive 2.3.9 client works well on communicating with the Hive Metastore server through 2.1 to 3.1 (maybe 2.0 too, not sure). This brings a new question, does the IsolatedClientLoader required then? > > I think we should drop IsolatedClientLoader because > > 1. As explained above, we can use the built-in Hive 2.3.9 client to communicate with HMS 2.1+ > 2. Since SPARK-42539[2], the default Hive 2.3.9 client does not use IsolatedClientLoader, and as explained in SPARK-42539, IsolatedClientLoader causes some inconsistent behaviors. > 3. It blocks Guava upgrading. HIVE-27560[3] aim to make Hive 2.3.10(unreleased) compatible with all Guava 14+ versions, but unfortunately, Guava is marked as `isSharedClass`[3] in IsolatedClientLoader, so technically, if we want to upgrade Guava we need to make all supported Hive versions(through 2.1.x to 3.1.x) to support high version of Guava, I think it's impossible. > > [1] sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientVersions.scala > [2] https://issues.apache.org/jira/browse/SPARK-42539 > [3] https://issues.apache.org/jira/browse/HIVE-27560 > [4] https://github.com/apache/spark/pull/33989#issuecomment-926277286 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
