+1 drop hadoop 2. On Thu, Jun 29, 2023 at 12:50 PM Xavier Léauté <xav...@confluent.io.invalid> wrote:
> +1, does this mean we would mark Hadoop 2 deprecated in Druid 27? > > Also, do we have a broader plan to remove Hadoop in general from core > dependencies and make a an optional extension? > > On Tue, Jun 27, 2023 at 11:53 PM Karan Kumar <karankumar1...@gmail.com> > wrote: > > > In favour of dropping hadoop 2 support . Another point is the lack of > > security and vulnerability fixes in hadoop2. > > > > > > > > On Wed, Jun 28, 2023 at 12:17 PM Clint Wylie <cwy...@apache.org> wrote: > > > > > obvious +1 from me > > > > > > On Tue, Jun 27, 2023 at 11:42 PM Gian Merlino <g...@apache.org> wrote: > > > > > > > > I'd like to propose dropping support for Hadoop 2 in Druid 28. Not > the > > > very > > > > next release (which I assume will be Druid 27) but the one after > that, > > > > likely late 2023 timeframe. > > > > > > > > In 2021, we had a discussion about moving away from Hadoop 2: > > > > https://lists.apache.org/thread/zmc389trnkh6x444so8mdb2h0x0noqq4. > For > > > > various reasons, it didn't seem like the right time. However, I > believe > > > now > > > > is the right time: > > > > > > > > 1) We didn't support Hadoop 3 in 2021, but we support it now. There > is > > > now > > > > a Hadoop 3 build profile, as well as convenience binaries on > > > > https://druid.apache.org/downloads.html. > > > > > > > > 2) We have SQL-based ingest with MSQ tasks, which provides a > built-in / > > > > scalable / robust alternative to using Hadoop at all. > > > > > > > > 3) It has been an additional two years. Hadoop 2 is that much older, > > that > > > > much more time has passed since it was superseded by Hadoop 3, and > > people > > > > have had that much more time to migrate. > > > > > > > > 4) The original main reason for wanting to move away from Hadoop 2 is > > > still > > > > relevant. It keeps us on various old dependencies, including an > ancient > > > > version of Guava, which in turn has been keeping us on an ancient > > version > > > > of Calcite. The Calcite community has graciously decided to support > > this > > > > old version of Guava for at least one release, but plans to drop > > support > > > by > > > > Calcite 1.36, leaving us back in the same position. Managing this > > > situation > > > > is time-consuming for both Druid and Calcite maintainers. > > > > > > > > 5) Other solutions beyond dropping Hadoop 2 support were proposed in > > > 2021, > > > > such as reworking Hadoop support to be purely extension based, and > > > > reworking extensions to be more isolated from each other. However, > > these > > > > are both substantially more complex than dropping support, and in the > > two > > > > years since the original thread, these more complex solutions have > not > > > been > > > > implemented. So, I think we need to move on with the simpler solution > > of > > > > dropping support. > > > > > > > > Gian > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org > > > For additional commands, e-mail: dev-h...@druid.apache.org > > > > > > > > > > -- > > Thanks > > Karan > > >
