Hi Tanvi,
Thanks! I will check that and have a talk with my colleagues to consider about the upgrading. Best, Zhefu Peng ------------------ ???????? ------------------ ??????: "Tanvi Thacker"<tanvithack...@gmail.com>; ????????: 2018??7??21??(??????) ????3:24 ??????: "user"<user@hive.apache.org>; ????: Re: Does Hive 3.0 only works with hadoop3.x.y? I would recommend upgrading to Hadoop 3.0 or 3.1 because of the following reasons:- It may be possible that Hadoop 2.x transitively brings some dependencies which may conflict with libraries used by hive( like unpredictable library google guava etc), which will affect your runtime environment. Hive might be utilizing some of the new public APIs which are exposed in 3.x line of Hadoop , so with Hadoop 2.x you may see some ClassNotFound/NoSuchMethod in runtime if your query is addressing such code path. In production, you must use the same the dependencies in which hive is compiled and tested. https://github.com/apache/hive/blob/rel/release-3.0.0/pom.xml#L149 Thanks, Tanvi Thacker On Thu, Jul 19, 2018 at 8:15 PM, Sungwoo Park <glap...@gmail.com> wrote: I would say yes (because I am actually running Hive 3.0 on Hadoop 2.7.6 and HDP 2.7.5), provided that you make small changes to the source code to Hive 3.0. However, I have not tested Hive 3.0 on Spark. --- Sungwoo On Thu, Jul 19, 2018 at 10:34 PM, ?????? <461292...@qq.com> wrote: Hi Sungwoo?? Just want to confirm, does that mean I just need to update the hive version, without updating the hadoop version? Thanks! Best, Zhefu Peng ------------------ ???????? ------------------ ??????: "Sungwoo Park"<glap...@gmail.com>; ????????: 2018??7??19??(??????) ????8:20 ??????: "user"<user@hive.apache.org>; ????: Re: Does Hive 3.0 only works with hadoop3.x.y? Hive 3.0 make a few function calls that depend on Hadoop 3.x, but they are easy to replace with code that compiles okay on Hadoop 2.8+. I am currently running Hadoop 3.x on Hadoop 2.7.6 and HDP 2.6.4 to test with the TPC-DS benchmark, and have not encountered any compatibility issue yet. I previously posted a diff file that lets us compile Hadoop 3.x on Hadoop 2.8+. http://mail-archives.apache.org/mod_mbox/hive-user/201806.mbox/%3CCAKHFPXDDFn52buKetHzSXTtjzX3UMHf%3DQvxm9QNNkv9r5xBs-Q%40mail.gmail.com%3E --- Sungwoo Park On Thu, Jul 19, 2018 at 8:21 PM, ?????? <461292...@qq.com> wrote: Hi, I already deployed hive 2.2.0 on our hadoop cluster. And recently, we deployed the spark cluster with 2.3.0, aiming at using the feature that hive on spark engine. However, when I checked the website of hive release, I found the text below: 21 May 2018 : release 3.0.0 available This release works with Hadoop 3.x.y. Now the hadoop version we deployed is hadoop 2.7.6. I wonder, does Hive 3.0 only work with hadoop 3.x.y? Or, if we want to use hive 3.0, we have to update the hadoop version to 3.x.y? Looking forward to your reply and help. Best, Zhefu Peng
