CC-ing to common-dev@hadoop.apache.org for more exposure. On Fri, Mar 7, 2025 at 8:02 AM Istvan Toth <st...@apache.org> wrote:
> Hi! > > I've been working on JDK23 support for a few weeks, and now I have a > mostly working patch (a few test failures may still happen). > > The existing HADOOP-19212 JIRA and its original PR is very incomplete, and > does not deal at all with the removal of Subject propagation into new > threads (which Hadoop heavily relies on) > > I have opened a new ticket HADOOP-19486 for the increased scope of the > JDK23 work, and linked my 99% working rough draft PR to it. > > I'm aware that getting such a huge change reviewed and committed is a huge > task, so I'd like to use this forum to discuss the changes, and how to > structure the work. > > The major changes in the patch: > > * Adds SubjectUtils as a JDK compatibility layer, (similar to the original > patch) > * Replaces deprecated direct deprecated Subject method calls in code > (including UGI) > * Adds callAs() to UserGroupInformation, and deprecates the old doAs() > calls. > * Adds a new HadoopThread wrapper class (and patches Daemon) to preserve > the pre-JEP411 Subject propagation semantics. > * Replaces all Thread class usage with HadoopThread. Unfortunately, the > only way I could get this to work was adding a new HadoopThread.work() > method which wraps the Thread.run(), and HadoopThread subclasses have to > override work() instead of run() (Runnable works without changes) > > Other than that there are a lot of smaller changes, like disabling tests > if they require JavaScript support and JDK does not have it, disabling > tests using SecurityManager if it is not available, adapting to toString() > output changes, not using 0.0.0.0 as target address, etc. > > The current patch is huge, a bit rough and incomplete, as I haven't > replaced all deprecated UGI calls. > > The test suite can be run with JDK23 by running *start-build-env.sh* and > setting *JAVA_HOME* to */usr/lib/jvm/temurin-23-jdk-amd64/*. (I have also > added JDK23 to the ARM image, but I could not test that) > > I am prepared to split this up to make reviewing more palatable, but I > would first like to get approval on how to do that. My current plan is: > - Add JDK23 support to Docker images > - Add all the small changes not directly related to UGI and > HadoopThread/Daemon > - Add the Subject and thread propagation related changes. (the difficult > parts) > - Replace newly deprecated calls in non-test code > - Replace newly deprecated calls in test code > - Add JDK23 or 24 runs to CI > > This should group the high impact changes into a single patch that is > testable and as small as possible > > Please review the *UserGroupInformation*, *SubjectUtil*, *HadoopThread* > and *Daemon* classes in the draft patch as those are the possibly > controversial changes, and indicate if the general solution is acceptable, > or if you have suggestions to make it better. > > Please also comment on the plan for the final patches. Is it acceptable ? > Do you have suggestions for improvements ? > > Istvan > >
