Regarding Hadoop: if core Druid code starts requiring Java 17, we might run into issues with running that core Druid code inside the remote Hadoop M/R processes. People would need to update their YARN runners to Java 17. And given Hadoop doesn't officially support Java 17 yet, this might cause problems with Hadoop itself. This set of challenges would I think be more troublesome than running the Hadoop client inside Druid processes.
To me this is a strong additional reason to stop supporting Hadoop sooner rather than later. The need for our code to be able to run inside Hadoop M/R processes, given how slow Hadoop moves, creates a need to support older Java versions and imposes a limit on our ability to use new Java features. Gian On 2024/12/17 07:44:12 Abhishek Agarwal wrote: > Do we really need to wait for Hadoop runtime to support Java 17 if the > Hadoop client jars themselves can be used in JDK 17 runtime? Spark dropped > support for Java 11 but I think, spark jobs can still use Hadoop client > code. So I am not sure if Hadoop is really a blocker for us to move off > Java 11. > > On Thu, Dec 12, 2024 at 2:08 AM Clint Wylie <cwy...@apache.org> wrote: > > > Now that we have removed support for Java 8, I wanted to start a > > discussion about dropping support for Java 11 as well since it is also > > pretty old, and making 17 the minimum supported version. There are a > > lot of nice language features with newer java versions, so getting a > > bit more aggressive about refreshing the minimum supported version > > periodically would allow us to begin to take advantage of these > > improvements, and would also reduce the number of tests we need to run > > in the CI pipeline. > > > > I am aware of a couple of things to consider in this discussion, the > > first being that Hadoop does not yet support Java 17 as a runtime. > > Though it does seem to be planned > > https://issues.apache.org/jira/browse/HADOOP-17177, I am unsure of the > > timeline for it to be released, so we might need to wait until this > > happens before we can totally remove it. I am starting another thread > > to survey Hadoop usage to see if this actually needs to be a blocker > > or not. > > > > The other thing I believe we would need to resolve is the Javascript > > based functionality, which is disabled by default in Druid, requires > > some work to keep being supported. > > https://github.com/apache/druid/pull/14795 describes the details I > > believe, and now that Java 8 has been dropped can probably be > > re-opened or at least used as a start to resolve this problem. > > > > Anyone aware of any additional issues with doing this? > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org > > For additional commands, e-mail: dev-h...@druid.apache.org > > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org For additional commands, e-mail: dev-h...@druid.apache.org
