pan3793 commented on PR #42493: URL: https://github.com/apache/spark/pull/42493#issuecomment-1687562520
@JoshRosen thanks for the comments, https://github.com/apache/spark/pull/42599 is opened to try removing Guava from `sharedClasses`, let's wait for CI results. > If Spark 4.0 drops JDK8 support then we'd already lose support for Hive < 2.0 using out-of-the-box Hive metastore clients. Yes, definitely, from the community perspective, that means Spark 4.0 drops support for HMS < 2.0. > However, some users run forked versions of old Hive which do support newer JDKs and for them the `IsolatedClientLoader` removal would create pains for their Spark 4.0 upgrade. Since those users are already forking Hive, though, I suppose they could simply recompile their Hive fork against Spark's newer Guava version. I think it's out of the scope of the Spark community's responsibility. And it's very weird to combine Hive 0.13(released in 2014) with Spark 4.0(will release in 2024), considering that Spark 4.0 decided to drop support for Java 8 and Hadoop 2, I don't think we should insist to support such an old Hive version. I definitely see lots of users fork and maintain their internal Hadoop/Hive/Spark versions, but as you said, they could simply recompile their Hive fork against Spark's newer Guava version, I suppose they also could recompile their Spark fork against their Hive fork. In short, I think it does not make sense that a user supposes Spark 4.0.0 should work out-of-box with a forked Hive released a decade ago. He/she should expect some code changes on Spark to make such a combination work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
