Hi all, > Also I have submitted PRs for first 2 tasks of phase one and are ready for > review: > * HBASE-29225 Add module for Jetty 12 with EE8 to hbase-thirdparty > * HBASE-29226 Migrate to jetty 12 with EE8 and bump java servlet to 4.0.1
Gentle ping, please review the following PRs. Istvan has helped in initial review of both of the PRs: * https://github.com/apache/hbase-thirdparty/pull/131 * https://github.com/apache/hbase/pull/6783 Please let us know if any other change is needed. Will wait a few more days before merging, if no further comments. Regards, Nihal On 2025/03/27 12:52:42 Nihal Jain wrote: > Hi > > I have created an umbrella ticket for Migrate from Jetty 9 to Jetty 12 at > HBASE-29224 > > Also I have submitted PRs for first 2 tasks of phase one and are ready for > review: > * HBASE-29225 Add module for Jetty 12 with EE8 to hbase-thirdparty > * HBASE-29226 Migrate to jetty 12 with EE8 and bump java servlet to 4.0.1 > > Have fixed most of the UTs locally. May need more testing: try SSL, and other > stuff; Will do as part of sub-task HBASE-29227. > > Also could someone please help/guide me with pushing the snapshot thirdparty > changes to artifactory to test out the main code and trigger UT without > merging? > > > For step 2 could we duplicate AuthenticationFilter ? > > We could do that but that would lead us to copy a lot of files. Alternately > we could temporarily create a shaded hadoop jar with relocated import > packages? Will need to see what is best once we get there. > > > Are the Jetty EE modules exclusive for the same namespace ? > > Jetty 12 with EE8 is the only module which supports javax. > Jetty 12 with EE9/EE10 strictly required jakarta. Please refer > https://stackoverflow.com/a/77010707/3778833 for exact matrix. > > Regards, > Nihal > > On 2025/03/12 06:57:40 Istvan Toth wrote: > > Sounds good. > > > > The 2.x backport still depends on bumping the minimum Java requirement. > > For step 2 could we duplicate AuthenticationFilter ? > > Are the Jetty EE modules exclusive for the same namespace ? > > > > Istvan > > > > On Tue, Mar 11, 2025 at 9:36 PM Nihal Jain <nihalj...@apache.org> wrote: > > > > > Hi all, > > > > > > Sorry for delay in responding. I have been spending some time trying to > > > get a feel of what all we might need to do by changing code (roughly), > > > hence thought to reply once I have something substantial ready. > > > > > > > If we could target EE 9 instead of EE 8 that would align with some > > > internal > > > > requirements at $dayjob. For what it's worth. Mostly the point of EE 9 > > > is a > > > > move of the EE stuff into a new namespace, allowing for further > > > development > > > > under the Eclipse Foundation without trademark conflicts with Oracle's > > > > "Java" branding. > > > > > > Sure we can aim for EE9 + jakarta. > > > > > > > I'd split that into two tickets. > > > > > > > > First move to Jetty 12 while keeping the old javax package names, and > > > once > > > > that's done and tested, we can look into moving to jakarta. > > > > > > But I think we should do this in 2 steps as mentioned by Istvan. > > > > > > As per me we should break the migration into below phases: > > > > > > Phase 1: > > > * Add module for Jetty 12 with EE8 to hbase-thirdparty > > > * Next consume this version of hbase-thirdparty, move to jetty 12 with EE8 > > > and bump java servlet to 4.0.1 > > > * Test and verify everything is working as expected. > > > * NOTE: We could also consider this as the fix we backport to branch-2 as > > > it seemingly does not change code/compat much. > > > > > > Phase 2: > > > * Add Jetty 12 with EE9 to hbase-thirdparty and jersey 3. And may be some > > > other artifacts (not sure at this point) > > > * Next consume this version of hbase-thirdparty, move to jetty 12 with > > > EE9, bump jakarta servlet to 5.x / 6.x, tomcat to 10.x / 11.x and migrate > > > all the dependencies and code to jakarta namespace > > > * Blockers?? Hadoop AuthenticationFilter dependent and related code need > > > to be either shaded to move from javax to jakarta, or we would need to > > > wait > > > for hadoop for move to jakarta. (In my rough analysis, I have identified > > > this till now, when we attempt it might be more stuff) > > > * Test and verify everything is working as expected. > > > > > > Update on phase 1: I have posted a draft working set in github. Have done > > > basic sanity of hbase, rest and thrift, looks good at high level. May need > > > more testing: try SSL, and other stuff. Have not tried UTs and other > > > things > > > yet. > > > > > > See draft PRs at: > > > * https://github.com/apache/hbase-thirdparty/pull/131 > > > * https://github.com/apache/hbase/pull/6783 > > > > > > Regards, > > > Nihal > > > > > > > > > References: > > > * > > > https://stackoverflow.com/questions/77007560/missing-jetty-servlet-12-0-0-dependency > > > * https://jetty.org/download.html#version-history > > > * https://tomcat.apache.org/whichversion.html > > > * > > > https://tomcat.apache.org/migration-10.html#Migrating_from_9.0.x_to_10.0.x > > > * > > > https://github.com/apache/hadoop/blob/branch-3.4.1/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java > > > > > > On 2025/03/07 17:13:41 Andrew Purtell wrote: > > > > I was not aware that Jetty 12 requires Java 17 but should have checked. > > > At some point we will need to concede to raising our minimum Java version > > > when a combination of security issues and dependency maintenance status > > > require the bump to solve for the constraints. We can try to get ahead of > > > it too. I think it makes sense. > > > > > > > > Let me look at PHOENIX-7164! > > > > > > > > > On Mar 7, 2025, at 7:35 AM, Istvan Toth <st...@cloudera.com.invalid> > > > wrote: > > > > > > > > > > The only way to get Jetty 17 into 2.x would be by bumping the > > > required Java > > > > > version for 2.7 or 2.8 to 17. > > > > > This is not necessarily out of the question, but we haven't discussed > > > that > > > > > yet. > > > > > > > > > > The main reason I pushed for bumping the minimum Java to 17 for 3.0 > > > was so > > > > > that we can move off all the old dependencies we are stuck on. > > > > > > > > > > BTW I heard noises (well, read it on ticket) that Hadoop 3.5 is also > > > > > planned to require JDK17. > > > > > > > > > > Aside: > > > > > Andrew, there are Phoenix PRs waiting for reviews which add the bulk > > > > > of > > > > > changes required for HBase 3.0 support. (Mostly just getting rid of > > > HBase > > > > > 1.0 APIs) > > > > > I'd appreciate it if someone from your team could review them. See > > > > > PHOENIX-7164. I'd like to add support for HBase 3.0 > > > > > to Phoenix promptly after 3.0 is released. > > > > > > > > > > Istvan > > > > > > > > > > > > > > >> On Fri, Mar 7, 2025 at 2:56 AM Andrew Purtell <apurt...@apache.org> > > > wrote: > > > > >> > > > > >> I'm not sure what other user appetites for HBase 3 in production are, > > > but > > > > >> at the $dayjob it's got to be 18 months to 2 years out. First, we are > > > a > > > > >> Phoenix shop, so Phoenix must adapt their code for HBase 3 and > > > > >> prepare > > > > >> releases based on HBase 3 that will be production ready, and as far > > > as I > > > > >> know this isn't on the radar for them yet. (It's quite possible our > > > shop > > > > >> would do the work, but not until next year at the earliest.) Then, we > > > must > > > > >> update all of our internal code for API differences, and rigorously > > > test > > > > >> incremental upgrade and rollback, until we can move up and back > > > seamlessly, > > > > >> and that will take time. I think we might consider allocating > > > resources for > > > > >> it beginning next year. I don't think we are unusual in that regard. > > > > >> > > > > >> All this to say, maintenance changes like a Jetty uplift should not > > > > >> be > > > > >> restricted to 3.x only, because 2.x users are going to remain on 2.x > > > for a > > > > >> while, and 2.x users have just as much exposure to security blockers > > > > >> related to end of support versions of Jetty. If the project doesn't > > > land > > > > >> things like Jetty uplift back to 2.x, that's just more work for us to > > > do > > > > >> it, and less bandwidth to pursue more forward looking activities. > > > > >> > > > > >> For your consideration. > > > > >> > > > > >> On Thu, Mar 6, 2025 at 5:45 PM 张铎(Duo Zhang) <palomino...@gmail.com> > > > > >> wrote: > > > > >> > > > > >>> If we can speed up the 3.x release, since it has already moved up > > > > >>> the > > > > >>> JDK17+, it will be easier to land these changes in 3.x only. > > > > >>> > > > > >>> Istvan Toth <st...@cloudera.com.invalid> 于2025年3月7日周五 03:42写道： > > > > >>>> > > > > >>>> I'd split that into two tickets. > > > > >>>> > > > > >>>> First move to Jetty 12 while keeping the old javax package names, > > > and > > > > >>> once > > > > >>>> that's done and tested, we can look into moving to jakarta. > > > > >>>> > > > > >>>> Istvan > > > > >>>> > > > > >>>> On Thu, Mar 6, 2025 at 8:33 PM Andrew Purtell <apurt...@apache.org> > > > > >>> wrote: > > > > >>>> > > > > >>>>>> Therefore, I plan to use the EE8 environment on Jetty 12. > > > > >>>>> > > > > >>>>> Ah, thanks for mentioning EE. > > > > >>>>> > > > > >>>>> Do you have a requirement to stay with EE 8 ? > > > > >>>>> > > > > >>>>> If we could target EE 9 instead of EE 8 that would align with some > > > > >>> internal > > > > >>>>> requirements at $dayjob. For what it's worth. Mostly the point of > > > EE > > > > >> 9 > > > > >>> is a > > > > >>>>> move of the EE stuff into a new namespace, allowing for further > > > > >>> development > > > > >>>>> under the Eclipse Foundation without trademark conflicts with > > > > >> Oracle's > > > > >>>>> "Java" branding. The trademark issue and follow on Java licensing > > > > >>> concerns > > > > >>>>> are going to be relevant for anyone who might be visited by an > > > Oracle > > > > >>>>> lawyer-shark. > > > > >>>>> > > > > >>>>> On Tue, Mar 4, 2025 at 6:42 AM Nihal Jain <nihalj...@apache.org> > > > > >>> wrote: > > > > >>>>> > > > > >>>>>>> I think that the best course of action would be adding a > > > > >>>>>>> separate > > > > >>>>> Jetty11 > > > > >>>>>> module to hbase-thirdparty, this way 9.4.x and 11.0.x can be both > > > > >>>>>> used/updated by the 2.x / 3.x branches. > > > > >>>>>> > > > > >>>>>> +1, I would like to volunteer for this work. > > > > >>>>>> > > > > >>>>>> Maintaining Jetty 9.4.x could be quite challenging from a CVE > > > > >>> perspective > > > > >>>>>> since it has been EOCS (End of Community Support) since June > > > > >>>>>> 2022, > > > > >>> and > > > > >>>>> new > > > > >>>>>> releases may not be forthcoming for a long time. I faced similar > > > > >>> issues > > > > >>>>>> during our last hbase-thirdparty release. See this issue comment: > > > > >>>>>> > > > > >>>>> > > > > >>> > > > > >> > > > https://github.com/jetty/jetty.project/issues/12630#issuecomment-2604701092 > > > > >>>>>> > > > > >>>>>> Regarding the version and CVEs, I agree with @Andrew and suggest > > > > >>> that we > > > > >>>>>> jump directly to Jetty 12, bypassing Jetty 11, to support > > > > >>> javax.servlet > > > > >>>>> for > > > > >>>>>> JSP 2.3.1. Therefore, I plan to use the EE8 environment on Jetty > > > > >> 12. > > > > >>> See > > > > >>>>>> Jetty version history: > > > > >>> https://jetty.org/download.html#version-history > > > > >>>>>> > > > > >>>>>> @Istvan, is there any particular reason you recommend moving to > > > > >>> Jetty 11, > > > > >>>>>> which is also EOCS? > > > > >>>>>> > > > > >>>>>> If others are fine with me taking this up, I can create the > > > > >> necessary > > > > >>>>>> JIRAs for the Jetty migration project and start the work soon. > > > > >>>>>> > > > > >>>>>> Regards, > > > > >>>>>> Nihal > > > > >>>>>> > > > > >>>>>> On 2025/03/03 18:08:53 Wei-Chiu Chuang wrote: > > > > >>>>>>> FYI I think Hadoop will need to make similar moves too. > > > > >>>>>>> Sounds like Hadoop should start the work to drop JDK8 as well. I > > > > >>> know > > > > >>>>>> Steve > > > > >>>>>>> has a patch that requires JDK17+ due to Iceberg > > > > >>>>>>> https://github.com/apache/hadoop/pull/7316 > > > > >>>>>>> > > > > >>>>>>> ---------- Forwarded message --------- > > > > >>>>>>> From: Istvan Toth <st...@apache.org> > > > > >>>>>>> Date: Sun, Mar 2, 2025 at 9:41 PM > > > > >>>>>>> Subject: [DISCUSS] Plans for JDK22 / SecurityManager removal > > > > >>>>>>> To: HBase Dev List <dev@hbase.apache.org> > > > > >>>>>>> > > > > >>>>>>> > > > > >>>>>>> Hi! > > > > >>>>>>> > > > > >>>>>>> The last big incompatible JDK change is the JEP411/JEP486 > > > > >>>>> SecurityManager > > > > >>>>>>> removal process, which has gotten serious with JDK22, which > > > > >>> disables > > > > >>>>>>> Subject.doAs*() Subject.getSubject() by default (while providing > > > > >>> the > > > > >>>>> kind > > > > >>>>>>> of equivalent callAs() and current() methods), and kills > > > > >> automatic > > > > >>>>>> Subject > > > > >>>>>>> propagation to new threads. > > > > >>>>>>> > > > > >>>>>>> To add insult to injury the new methods are only available from > > > > >>> JDK18, > > > > >>>>> so > > > > >>>>>>> it is not possible to just move to the new API without breaking > > > > >>> JDK17. > > > > >>>>>>> > > > > >>>>>>> Some of this can only be solved in Hadoop (hopefully 3.5) , but > > > > >>> HBase > > > > >>>>> can > > > > >>>>>>> also take steps to move towards JDK22 compatibility in the > > > > >>> meantime. > > > > >>>>>>> > > > > >>>>>>> Here's the plan I have in mind: > > > > >>>>>>> > > > > >>>>>>> - Review and the dependencies WRT SecurityManage usage (just > > > > >>> search for > > > > >>>>>> API > > > > >>>>>>> calls) > > > > >>>>>>> - Update the dependencies as needed > > > > >>>>>>> > > > > >>>>>>> - Add a reflection-based compatibility Utility class based on > > > > >>> Jetty's > > > > >>>>>>> SecurityUtils, (or one of its derivatives) > > > > >>>>>>> - Replace the deprecated method calls with the compatibility > > > > >>> versions > > > > >>>>>>> - Wrap Runnables/Callables with Subject current()/callAs() > > > > >> methods. > > > > >>>>>>> > > > > >>>>>>> One large dependency I have identified is Jetty. > > > > >>>>>>> While technically we MAY be able to keep using 9.4 by overriding > > > > >>> its > > > > >>>>>>> default ThreadFactory, I think that considering that even > > > > >>> semi-formal > > > > >>>>>> Jetty > > > > >>>>>>> 9.4 support is on its very last legs (the latest update was > > > > >>>>>>> never > > > > >>> even > > > > >>>>>>> announced and is not listed in the downloads page), the > > > > >>> advantages of > > > > >>>>>>> updating to Jetty 11 on branch-3+ now far outweighs the added > > > > >>>>> maintenance > > > > >>>>>>> burden of the branch divergence. > > > > >>>>>>> > > > > >>>>>>> I think that the best course of action would be adding a > > > > >>>>>>> separate > > > > >>>>> Jetty11 > > > > >>>>>>> module to hbase-thirdparty, this way 9.4.x and 11.0.x can be > > > > >>>>>>> both > > > > >>>>>>> used/updated by the 2.x / 3.x branches. > > > > >>>>>>> > > > > >>>>>>> What do you think ? > > > > >>>>>>> > > > > >>>>>>> - Do you agree that JEP411/486 should be treated as a priority ? > > > > >>>>>>> - Would you volunteer for the dependency audit ? > > > > >>>>>>> - Do you agree with the Jetty update, and the plan outlined > > > > >> above ? > > > > >>>>>>> > > > > >>>>>>> Istvan > > > > >>>>>>> > > > > >>>>>>> ps: Sorry, no links because GMail seems to react very bady to > > > > >> any. > > > > >>>>> Please > > > > >>>>>>> use search. > > > > >>>>>>> > > > > >>>>>> > > > > >>>>>> > > > > >> --------------------------------------------------------------------- > > > > >>>>>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > > > > >>>>>> For additional commands, e-mail: > > > common-dev-h...@hadoop.apache.org > > > > >>>>>> > > > > >>>>>> > > > > >>>>> > > > > >>>>> -- > > > > >>>>> Best regards, > > > > >>>>> Andrew > > > > >>>>> > > > > >>>>> Unrest, ignorance distilled, nihilistic imbeciles - > > > > >>>>> It's what we’ve earned > > > > >>>>> Welcome, apocalypse, what’s taken you so long? > > > > >>>>> Bring us the fitting end that we’ve been counting on > > > > >>>>> - A23, Welcome, Apocalypse > > > > >>>>> > > > > >>>> > > > > >>>> > > > > >>>> -- > > > > >>>> *István Tóth* | Sr. Staff Software Engineer > > > > >>>> *Email*: st...@cloudera.com > > > > >>>> cloudera.com <https://www.cloudera.com> > > > > >>>> [image: Cloudera] <https://www.cloudera.com/> > > > > >>>> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > > > > >>>> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > > > > >>> Cloudera > > > > >>>> on LinkedIn] <https://www.linkedin.com/company/cloudera> > > > > >>>> ------------------------------ > > > > >>>> ------------------------------ > > > > >>> > > > > >> > > > > >> > > > > >> -- > > > > >> Best regards, > > > > >> Andrew > > > > >> > > > > >> Unrest, ignorance distilled, nihilistic imbeciles - > > > > >> It's what we’ve earned > > > > >> Welcome, apocalypse, what’s taken you so long? > > > > >> Bring us the fitting end that we’ve been counting on > > > > >> - A23, Welcome, Apocalypse > > > > >> > > > > > > > > > > > > > > > -- > > > > > *István Tóth* | Sr. Staff Software Engineer > > > > > *Email*: st...@cloudera.com > > > > > cloudera.com <https://www.cloudera.com> > > > > > [image: Cloudera] <https://www.cloudera.com/> > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > > > Cloudera > > > > > on LinkedIn] <https://www.linkedin.com/company/cloudera> > > > > > ------------------------------ > > > > > ------------------------------ > > > > > > > > > > > > > -- > > *István Tóth* | Sr. Staff Software Engineer > > *Email*: st...@cloudera.com > > cloudera.com <https://www.cloudera.com> > > [image: Cloudera] <https://www.cloudera.com/> > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera > > on LinkedIn] <https://www.linkedin.com/company/cloudera> > > ------------------------------ > > ------------------------------ > > >
