I don't want to kick that can too far down the road either :) We don't want to give a false hope that it's going to remain around forever. But yes let's deprecate both Hadoop and Java 11 support in the upcoming 32 release. It's unfortunate that Hadoop still doesn't support Java 17. We shouldn't let it hold us back. Jetty, pac4j are dropping Java 11 support and we would want to upgrade to newer versions of these dependencies soon. There are also nice language features in Java 17 such as pattern matching, multiline strings, and a lot more that we can't use if we have to be compile compatible with Java 11. If you need the resource elasticity that Hadoop provides or want to reuse shared infrastructure in the company, MM-less ingestion is a good alternative.
So let's deprecate it in 32. We can decide on removal later but hopefully, it doesn't take too many releases to do that. On Tue, Jan 7, 2025 at 4:22 PM Karan Kumar <ka...@apache.org> wrote: > Okay from what I can gather few folks still need hadoop ingestion. So let's > kick the can down the road regarding removal of that support but let's > agree on the deprecation plan. Since druid 32 is around the corner let's > atleast deprecated hadoop ingestion so that any new users are not onboarded > to this way of ingestion. Deprecation also becomes a forcing function in > internal company channel's for prioritization of getting off hadoop. > > How does this plan look? > > On Fri, Dec 13, 2024 at 1:11 AM Maytas Monsereenusorn <mayt...@apache.org> > wrote: > > > We at Netflix are in a similar situation to Target Corporation (Lucas C > > email above). > > We currently rely on Hadoop ingestion for all our batch ingestion jobs. > The > > main reason for this is that we already have a large Hadoop cluster > > supporting our Spark workloads that we can leverage for Druid ingestion. > I > > imagine that the closest alternative for us would be moving to K8 / > > MiddleManager-less ingestion job. > > > > On Thu, Dec 12, 2024 at 10:56 PM Lucas Capistrant < > > capistrant.lu...@gmail.com> wrote: > > > > > Apologies for the empty email… fat fingers. > > > > > > Just wanted to say that we at Target Corporation (USA), still rely > > heavily > > > on Hadoop ingest. We’d selfishly want support forever, but if forced to > > > pivot to a new ingestion style for our larger batch ingest jobs that > > > currently leverage the cheap compute on YARN, the longer the lead time > > > between announcement by the community to the actual release with no > > > support, the better. Making these types of changes can be a slow > process > > > for the slow to maneuver corporate cruise ship. > > > > > > On Thu, Dec 12, 2024 at 9:46 AM Lucas Capistrant < > > > capistrant.lu...@gmail.com> > > > wrote: > > > > > > > > > > > > > > > On Wed, Dec 11, 2024 at 9:10 PM Karan Kumar <ka...@apache.org> > wrote: > > > > > > > >> +1 for removal of Hadoop based ingestion. It's a maintenance > overhead > > > and > > > >> stops us from moving to java 17. > > > >> I am not aware of any gaps in sql based ingestion which limits users > > to > > > >> move off from hadoop. If there are any, please feel free to reach > out > > > via > > > >> slack/github. > > > >> > > > >> On Thu, Dec 12, 2024 at 3:22 AM Clint Wylie <cwy...@apache.org> > > wrote: > > > >> > > > >> > Hey everyone, > > > >> > > > > >> > It is about that time again to take a pulse on how commonly Hadoop > > > >> > based ingestion is used with Druid in order to determine if we > > should > > > >> > keep supporting it or not going forward. > > > >> > > > > >> > In my view, Hadoop based ingestion has unofficially been on life > > > >> > support for quite some time as we do not really go out of our way > to > > > >> > add new features to it, and we perform very minimal testing to > > ensure > > > >> > everything keeps working. The most recent changes to it I am aware > > of > > > >> > was to bump versions and require Hadoop 3, but that was primarily > > > >> > motivated by selfish reasons of wanting to use its contained > client > > > >> > library and better isolation so that we could free up our own > > > >> > dependencies to be updated. This thread is motivated by a similar > > > >> > reason I guess, see the other thread I started recently discussing > > > >> > dropping support for Java 11 where Hadoop does not yet support > Java > > 17 > > > >> > runtime, and so the outcome of this discussion is involved in > those > > > >> > plans. > > > >> > > > > >> > I think SQL based ingestion with the multi-stage query engine is > the > > > >> > future of batch ingestion, and the Kubernetes based task runner > > > >> > provides an alternative for task auto scaling capabilities. > Because > > of > > > >> > this, I don't personally see a lot of compelling reasons to keep > > > >> > supporting Hadoop, so I would be in favor of just dropping support > > for > > > >> > it completely, though I see no harm in keeping HDFS deep storage > > > >> > around. In past discussions I think we had tied Hadoop removal to > > > >> > adding something like Spark to replace it, but I wonder if this > > still > > > >> > needs to be the case. > > > >> > > > > >> > I do know that classically there have been quite a lot of large > > Druid > > > >> > clusters in the wild still relying on Hadoop in previous dev list > > > >> > discussions about this topic, so I wanted to check to see if this > is > > > >> > still true and if so if any of these clusters have plans to > > transition > > > >> > to newer ways of ingesting data like SQL based ingestion. While > > from a > > > >> > dev/maintenance perspective it would be best to just drop it > > > >> > completely, if there is still a large user base I think we need to > > be > > > >> > open to keeping it around for a while longer. If we do need to > keep > > > >> > it, maybe it would be worth it to invest some time in moving it > > into a > > > >> > contrib extension so that it isn't bundled by default with Druid > > > >> > releases to discourage new adoption and more accurately reflect > its > > > >> > current status in Druid. > > > >> > > > > >> > > > --------------------------------------------------------------------- > > > >> > To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org > > > >> > For additional commands, e-mail: dev-h...@druid.apache.org > > > >> > > > > >> > > > > >> > > > > > > > > > >
