Re: Hive 3.1.3 Hadoop Compatability
BigTop supports a specific version stack with some patches in place. It should be helpful for you. - Currently, the master branch consists of Hadoop-3.3.6, Hive-3.1.3, Tez-0.10.2. - https://github.com/apache/bigtop/blob/master/bigtop.bom - Hive patches: https://github.com/apache/bigtop/tree/master/bigtop-packages/src/common/hive - Tez patches: https://github.com/apache/bigtop/tree/master/bigtop-packages/src/common/tez Disclaimer: I am not the developer of Hive/Tez/BigTop. Thanks, - Takanobu 2023年12月22日(金) 23:14 Austin Hackett : > Many thanks for clarifying Ayush - much appreciated > > > On 22 Dec 2023, at 08:41, Ayush Saxena wrote: > > > > Ideally the hadoop should be on 3.1.0 only, that is what we support, > > rest if there are no incompatibilities it might or might not work with > > higher versions of hadoop, we at "hive" don't claim that it can work, > > mostly it will create issues with hadoop-3.3.x line due to thirdparty > > libs and stuff like that, Guava IIRC does create some mess. > > > > So, short answer: we officially only support the above said hadoop > > versions only for a particular hive release. > > > > -Ayush > > > >> On Fri, 22 Dec 2023 at 03:03, Austin Hackett > wrote: > >> > >> Hi Ayush > >> > >> Many thanks for your response. > >> > >> I’d really appreciate a clarification if that’s OK? > >> > >> Does this just mean that the Hadoop 3.1.0 libraries need to be deployed > with Hive, or does it also mean the Hadoop cluster itself cannot be on a > version later than 3.1.0 (if using Hive 3.1.3). > >> > >> For example, if running the Hive 3.1.3 Metastore in standalone mode, > can the HMS work with a 3.3.6 HDFS cluster providing the Hadoop 3.1.0 > libraries are deployed alongside the HMS? > >> > >> Any help is much appreciated > >> > >> Thank you > >> > >> > >> > On 21 Dec 2023, at 12:18, Ayush Saxena wrote: > >>> > >>> Hi Austin, > >>> Hive 3.1.3 & 4.0.0-alpha-1 works with Hadoop-3.1.0 > >>> > >>> HIve 4.0.0-alpha-2 & 4.0.0-beta-1 works with Hadoop-3.3.1 > >>> > >>> The upcoming Hive 4.0 GA release would be compatible with Hadoop-3.3.6 > >>> > >>> -Ayush > >>> > >>> On Thu, 21 Dec 2023 at 17:39, Austin Hackett > wrote: > > Hi List > > I was hoping that someone might be able to clarify which Hadoop > versions Hive 3.1.3 is compatible with? > > https://hive.apache.org/general/downloads/ says that Hive release > 3.1.3 works with Hadoop 3.x.y which is straightforward enough. > > However, I notice the 4.0.0 releases only work with Hadoop 3.3.1, > which makes we wonder if 3.1.3 doesn’t work actually work with 3.3.1. > > Similarly, I see that HIVE-27757 upgrades Hadoop to 3.3.6 in Hive > 4.0.0, which makes me wonder if Hive 4.0.0 actually works with 3.3.6 and > not 3.3.1 as mentioned on the releases page. > > In summary: does Hive 3.1.3 work with Hadoop 3.3.6, and if not, which > Hadoop 3.x.x versions are known to work? > > Any pointers would be greatly appreciated > > Thank you > >> >
Re: Hive 3.1.3 Hadoop Compatability
Many thanks for clarifying Ayush - much appreciated > On 22 Dec 2023, at 08:41, Ayush Saxena wrote: > > Ideally the hadoop should be on 3.1.0 only, that is what we support, > rest if there are no incompatibilities it might or might not work with > higher versions of hadoop, we at "hive" don't claim that it can work, > mostly it will create issues with hadoop-3.3.x line due to thirdparty > libs and stuff like that, Guava IIRC does create some mess. > > So, short answer: we officially only support the above said hadoop > versions only for a particular hive release. > > -Ayush > >> On Fri, 22 Dec 2023 at 03:03, Austin Hackett wrote: >> >> Hi Ayush >> >> Many thanks for your response. >> >> I’d really appreciate a clarification if that’s OK? >> >> Does this just mean that the Hadoop 3.1.0 libraries need to be deployed with >> Hive, or does it also mean the Hadoop cluster itself cannot be on a version >> later than 3.1.0 (if using Hive 3.1.3). >> >> For example, if running the Hive 3.1.3 Metastore in standalone mode, can the >> HMS work with a 3.3.6 HDFS cluster providing the Hadoop 3.1.0 libraries are >> deployed alongside the HMS? >> >> Any help is much appreciated >> >> Thank you >> >> >> On 21 Dec 2023, at 12:18, Ayush Saxena wrote: >>> >>> Hi Austin, >>> Hive 3.1.3 & 4.0.0-alpha-1 works with Hadoop-3.1.0 >>> >>> HIve 4.0.0-alpha-2 & 4.0.0-beta-1 works with Hadoop-3.3.1 >>> >>> The upcoming Hive 4.0 GA release would be compatible with Hadoop-3.3.6 >>> >>> -Ayush >>> >>> On Thu, 21 Dec 2023 at 17:39, Austin Hackett wrote: Hi List I was hoping that someone might be able to clarify which Hadoop versions Hive 3.1.3 is compatible with? https://hive.apache.org/general/downloads/ says that Hive release 3.1.3 works with Hadoop 3.x.y which is straightforward enough. However, I notice the 4.0.0 releases only work with Hadoop 3.3.1, which makes we wonder if 3.1.3 doesn’t work actually work with 3.3.1. Similarly, I see that HIVE-27757 upgrades Hadoop to 3.3.6 in Hive 4.0.0, which makes me wonder if Hive 4.0.0 actually works with 3.3.6 and not 3.3.1 as mentioned on the releases page. In summary: does Hive 3.1.3 work with Hadoop 3.3.6, and if not, which Hadoop 3.x.x versions are known to work? Any pointers would be greatly appreciated Thank you >>
Re: Hive 3.1.3 Hadoop Compatability
Ideally the hadoop should be on 3.1.0 only, that is what we support, rest if there are no incompatibilities it might or might not work with higher versions of hadoop, we at "hive" don't claim that it can work, mostly it will create issues with hadoop-3.3.x line due to thirdparty libs and stuff like that, Guava IIRC does create some mess. So, short answer: we officially only support the above said hadoop versions only for a particular hive release. -Ayush On Fri, 22 Dec 2023 at 03:03, Austin Hackett wrote: > > Hi Ayush > > Many thanks for your response. > > I’d really appreciate a clarification if that’s OK? > > Does this just mean that the Hadoop 3.1.0 libraries need to be deployed with > Hive, or does it also mean the Hadoop cluster itself cannot be on a version > later than 3.1.0 (if using Hive 3.1.3). > > For example, if running the Hive 3.1.3 Metastore in standalone mode, can the > HMS work with a 3.3.6 HDFS cluster providing the Hadoop 3.1.0 libraries are > deployed alongside the HMS? > > Any help is much appreciated > > Thank you > > > > > On 21 Dec 2023, at 12:18, Ayush Saxena wrote: > > > > Hi Austin, > > Hive 3.1.3 & 4.0.0-alpha-1 works with Hadoop-3.1.0 > > > > HIve 4.0.0-alpha-2 & 4.0.0-beta-1 works with Hadoop-3.3.1 > > > > The upcoming Hive 4.0 GA release would be compatible with Hadoop-3.3.6 > > > > -Ayush > > > > On Thu, 21 Dec 2023 at 17:39, Austin Hackett wrote: > >> > >> Hi List > >> > >> I was hoping that someone might be able to clarify which Hadoop versions > >> Hive 3.1.3 is compatible with? > >> > >> https://hive.apache.org/general/downloads/ says that Hive release 3.1.3 > >> works with Hadoop 3.x.y which is straightforward enough. > >> > >> However, I notice the 4.0.0 releases only work with Hadoop 3.3.1, which > >> makes we wonder if 3.1.3 doesn’t work actually work with 3.3.1. > >> > >> Similarly, I see that HIVE-27757 upgrades Hadoop to 3.3.6 in Hive 4.0.0, > >> which makes me wonder if Hive 4.0.0 actually works with 3.3.6 and not > >> 3.3.1 as mentioned on the releases page. > >> > >> In summary: does Hive 3.1.3 work with Hadoop 3.3.6, and if not, which > >> Hadoop 3.x.x versions are known to work? > >> > >> Any pointers would be greatly appreciated > >> > >> Thank you >
Re: Hive 3.1.3 Hadoop Compatability
Hi Ayush Many thanks for your response. I’d really appreciate a clarification if that’s OK? Does this just mean that the Hadoop 3.1.0 libraries need to be deployed with Hive, or does it also mean the Hadoop cluster itself cannot be on a version later than 3.1.0 (if using Hive 3.1.3). For example, if running the Hive 3.1.3 Metastore in standalone mode, can the HMS work with a 3.3.6 HDFS cluster providing the Hadoop 3.1.0 libraries are deployed alongside the HMS? Any help is much appreciated Thank you > On 21 Dec 2023, at 12:18, Ayush Saxena wrote: > > Hi Austin, > Hive 3.1.3 & 4.0.0-alpha-1 works with Hadoop-3.1.0 > > HIve 4.0.0-alpha-2 & 4.0.0-beta-1 works with Hadoop-3.3.1 > > The upcoming Hive 4.0 GA release would be compatible with Hadoop-3.3.6 > > -Ayush > > On Thu, 21 Dec 2023 at 17:39, Austin Hackett wrote: >> >> Hi List >> >> I was hoping that someone might be able to clarify which Hadoop versions >> Hive 3.1.3 is compatible with? >> >> https://hive.apache.org/general/downloads/ says that Hive release 3.1.3 >> works with Hadoop 3.x.y which is straightforward enough. >> >> However, I notice the 4.0.0 releases only work with Hadoop 3.3.1, which >> makes we wonder if 3.1.3 doesn’t work actually work with 3.3.1. >> >> Similarly, I see that HIVE-27757 upgrades Hadoop to 3.3.6 in Hive 4.0.0, >> which makes me wonder if Hive 4.0.0 actually works with 3.3.6 and not 3.3.1 >> as mentioned on the releases page. >> >> In summary: does Hive 3.1.3 work with Hadoop 3.3.6, and if not, which Hadoop >> 3.x.x versions are known to work? >> >> Any pointers would be greatly appreciated >> >> Thank you
Re: Hive 3.1.3 Hadoop Compatability
Hi Austin, Hive 3.1.3 & 4.0.0-alpha-1 works with Hadoop-3.1.0 HIve 4.0.0-alpha-2 & 4.0.0-beta-1 works with Hadoop-3.3.1 The upcoming Hive 4.0 GA release would be compatible with Hadoop-3.3.6 -Ayush On Thu, 21 Dec 2023 at 17:39, Austin Hackett wrote: > > Hi List > > I was hoping that someone might be able to clarify which Hadoop versions Hive > 3.1.3 is compatible with? > > https://hive.apache.org/general/downloads/ says that Hive release 3.1.3 works > with Hadoop 3.x.y which is straightforward enough. > > However, I notice the 4.0.0 releases only work with Hadoop 3.3.1, which makes > we wonder if 3.1.3 doesn’t work actually work with 3.3.1. > > Similarly, I see that HIVE-27757 upgrades Hadoop to 3.3.6 in Hive 4.0.0, > which makes me wonder if Hive 4.0.0 actually works with 3.3.6 and not 3.3.1 > as mentioned on the releases page. > > In summary: does Hive 3.1.3 work with Hadoop 3.3.6, and if not, which Hadoop > 3.x.x versions are known to work? > > Any pointers would be greatly appreciated > > Thank you
