BigTop supports a specific version stack with some patches in place. It
should be helpful for you.
- Currently, the master branch consists of Hadoop-3.3.6, Hive-3.1.3,
Tez-0.10.2.
- https://github.com/apache/bigtop/blob/master/bigtop.bom
- Hive patches:
https://github.com/apache/bigtop/tree/master/bigtop-packages/src/common/hive
- Tez patches:
https://github.com/apache/bigtop/tree/master/bigtop-packages/src/common/tez
Disclaimer: I am not the developer of Hive/Tez/BigTop.
Thanks,
- Takanobu
2023年12月22日(金) 23:14 Austin Hackett <hacketta...@me.com>:
> Many thanks for clarifying Ayush - much appreciated
>
> > On 22 Dec 2023, at 08:41, Ayush Saxena <ayush...@gmail.com> wrote:
> >
> > Ideally the hadoop should be on 3.1.0 only, that is what we support,
> > rest if there are no incompatibilities it might or might not work with
> > higher versions of hadoop, we at "hive" don't claim that it can work,
> > mostly it will create issues with hadoop-3.3.x line due to thirdparty
> > libs and stuff like that, Guava IIRC does create some mess.
> >
> > So, short answer: we officially only support the above said hadoop
> > versions only for a particular hive release.
> >
> > -Ayush
> >
> >> On Fri, 22 Dec 2023 at 03:03, Austin Hackett <hacketta...@me.com>
> wrote:
> >>
> >> Hi Ayush
> >>
> >> Many thanks for your response.
> >>
> >> I’d really appreciate a clarification if that’s OK?
> >>
> >> Does this just mean that the Hadoop 3.1.0 libraries need to be deployed
> with Hive, or does it also mean the Hadoop cluster itself cannot be on a
> version later than 3.1.0 (if using Hive 3.1.3).
> >>
> >> For example, if running the Hive 3.1.3 Metastore in standalone mode,
> can the HMS work with a 3.3.6 HDFS cluster providing the Hadoop 3.1.0
> libraries are deployed alongside the HMS?
> >>
> >> Any help is much appreciated
> >>
> >> Thank you
> >>
> >>
> >>
> >>>> On 21 Dec 2023, at 12:18, Ayush Saxena <ayush...@gmail.com> wrote:
> >>>
> >>> Hi Austin,
> >>> Hive 3.1.3 & 4.0.0-alpha-1 works with Hadoop-3.1.0
> >>>
> >>> HIve 4.0.0-alpha-2 & 4.0.0-beta-1 works with Hadoop-3.3.1
> >>>
> >>> The upcoming Hive 4.0 GA release would be compatible with Hadoop-3.3.6
> >>>
> >>> -Ayush
> >>>
> >>> On Thu, 21 Dec 2023 at 17:39, Austin Hackett <hacketta...@me.com>
> wrote:
> >>>>
> >>>> Hi List
> >>>>
> >>>> I was hoping that someone might be able to clarify which Hadoop
> versions Hive 3.1.3 is compatible with?
> >>>>
> >>>> https://hive.apache.org/general/downloads/ says that Hive release
> 3.1.3 works with Hadoop 3.x.y which is straightforward enough.
> >>>>
> >>>> However, I notice the 4.0.0 releases only work with Hadoop 3.3.1,
> which makes we wonder if 3.1.3 doesn’t work actually work with 3.3.1.
> >>>>
> >>>> Similarly, I see that HIVE-27757 upgrades Hadoop to 3.3.6 in Hive
> 4.0.0, which makes me wonder if Hive 4.0.0 actually works with 3.3.6 and
> not 3.3.1 as mentioned on the releases page.
> >>>>
> >>>> In summary: does Hive 3.1.3 work with Hadoop 3.3.6, and if not, which
> Hadoop 3.x.x versions are known to work?
> >>>>
> >>>> Any pointers would be greatly appreciated
> >>>>
> >>>> Thank you
> >>
>
