amanraj2520 commented on PR #4025: URL: https://github.com/apache/hive/pull/4025#issuecomment-1421130592
@ayushtkn You are right when we try to integrate it with Tez, things can break. I agree fully (as you pointed out). **But in that case, we can ignore this commit in Hive 4.x release.** We can still have this PR merged to Hive master I think. The failure you are talking about is because Tez 0.10.2 uses 3.3.1 Hadoop and Hive will use 3.3.4 if this PR goes through. This is because Tez 0.10.2 brings in certain older jars in the classpath while a Hive job is executed as transitive dependencies for Hadoop 3.3.1. This was exactly why I was pushing for Tez 0.10.3 release which will have Hadoop 3.3.4 as we were discussing on this thread - https://github.com/apache/tez/pull/213. Personally, I have added internal commits in Tez 0.10.2 just for testing in my local cluster where I just upgraded Hadoop to 3.3.4 (in Tez 0.10.2) and used that SNAPSHOT jar in the Hive master pom. Those errors were resolved. So basically, what I am again suggesting is that we can have a Tez 0.10.3 release (with Hadoop 3.3.4), if everyone agrees. So that would mean Hive master, Tez 0.10.3 and Hadoop 3.3.4 can form a proper stack for end to end integration. Also, this will help in the ongoing hive-3.2.0 release that we are putting our efforts to where we are trying to use Hadoop 3.3.4 as the latest stack. Please let me know your opinion. cc @zabetak @abstractdog -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
