On Thu, Mar 27, 2025 at 1:52 PM Nihal Jain <nihalj...@apache.org> wrote: > > Also could someone please help/guide me with pushing the snapshot thirdparty > changes to artifactory to test out the main code and trigger UT without > merging?
You can configure your maven settings.xml file with your apache credentials for the snapshot server. You can then use maven deploy to publish a local build. I have not done this recently, so I'm afraid I don't have specifics to share. Maybe we should have our nightly jobs publish snapshots out of CI? I believe that we do not do this today because of challenges of credentials management, but that should be a solvable problem. Good on you Nihal. Thanks, Nick > On 2025/03/12 06:57:40 Istvan Toth wrote: > > Sounds good. > > > > The 2.x backport still depends on bumping the minimum Java requirement. > > For step 2 could we duplicate AuthenticationFilter ? > > Are the Jetty EE modules exclusive for the same namespace ? > > > > Istvan > > > > On Tue, Mar 11, 2025 at 9:36 PM Nihal Jain <nihalj...@apache.org> wrote: > > > > > Hi all, > > > > > > Sorry for delay in responding. I have been spending some time trying to > > > get a feel of what all we might need to do by changing code (roughly), > > > hence thought to reply once I have something substantial ready. > > > > > > > If we could target EE 9 instead of EE 8 that would align with some > > > internal > > > > requirements at $dayjob. For what it's worth. Mostly the point of EE 9 > > > is a > > > > move of the EE stuff into a new namespace, allowing for further > > > development > > > > under the Eclipse Foundation without trademark conflicts with Oracle's > > > > "Java" branding. > > > > > > Sure we can aim for EE9 + jakarta. > > > > > > > I'd split that into two tickets. > > > > > > > > First move to Jetty 12 while keeping the old javax package names, and > > > once > > > > that's done and tested, we can look into moving to jakarta. > > > > > > But I think we should do this in 2 steps as mentioned by Istvan. > > > > > > As per me we should break the migration into below phases: > > > > > > Phase 1: > > > * Add module for Jetty 12 with EE8 to hbase-thirdparty > > > * Next consume this version of hbase-thirdparty, move to jetty 12 with EE8 > > > and bump java servlet to 4.0.1 > > > * Test and verify everything is working as expected. > > > * NOTE: We could also consider this as the fix we backport to branch-2 as > > > it seemingly does not change code/compat much. > > > > > > Phase 2: > > > * Add Jetty 12 with EE9 to hbase-thirdparty and jersey 3. And may be some > > > other artifacts (not sure at this point) > > > * Next consume this version of hbase-thirdparty, move to jetty 12 with > > > EE9, bump jakarta servlet to 5.x / 6.x, tomcat to 10.x / 11.x and migrate > > > all the dependencies and code to jakarta namespace > > > * Blockers?? Hadoop AuthenticationFilter dependent and related code need > > > to be either shaded to move from javax to jakarta, or we would need to > > > wait > > > for hadoop for move to jakarta. (In my rough analysis, I have identified > > > this till now, when we attempt it might be more stuff) > > > * Test and verify everything is working as expected. > > > > > > Update on phase 1: I have posted a draft working set in github. Have done > > > basic sanity of hbase, rest and thrift, looks good at high level. May need > > > more testing: try SSL, and other stuff. Have not tried UTs and other > > > things > > > yet. > > > > > > See draft PRs at: > > > * https://github.com/apache/hbase-thirdparty/pull/131 > > > * https://github.com/apache/hbase/pull/6783 > > > > > > Regards, > > > Nihal > > > > > > > > > References: > > > * > > > https://stackoverflow.com/questions/77007560/missing-jetty-servlet-12-0-0-dependency > > > * https://jetty.org/download.html#version-history > > > * https://tomcat.apache.org/whichversion.html > > > * > > > https://tomcat.apache.org/migration-10.html#Migrating_from_9.0.x_to_10.0.x > > > * > > > https://github.com/apache/hadoop/blob/branch-3.4.1/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java > > > > > > On 2025/03/07 17:13:41 Andrew Purtell wrote: > > > > I was not aware that Jetty 12 requires Java 17 but should have checked. > > > At some point we will need to concede to raising our minimum Java version > > > when a combination of security issues and dependency maintenance status > > > require the bump to solve for the constraints. We can try to get ahead of > > > it too. I think it makes sense. > > > > > > > > Let me look at PHOENIX-7164! > > > > > > > > > On Mar 7, 2025, at 7:35 AM, Istvan Toth <st...@cloudera.com.invalid> > > > wrote: > > > > > > > > > > The only way to get Jetty 17 into 2.x would be by bumping the > > > required Java > > > > > version for 2.7 or 2.8 to 17. > > > > > This is not necessarily out of the question, but we haven't discussed > > > that > > > > > yet. > > > > > > > > > > The main reason I pushed for bumping the minimum Java to 17 for 3.0 > > > was so > > > > > that we can move off all the old dependencies we are stuck on. > > > > > > > > > > BTW I heard noises (well, read it on ticket) that Hadoop 3.5 is also > > > > > planned to require JDK17. > > > > > > > > > > Aside: > > > > > Andrew, there are Phoenix PRs waiting for reviews which add the bulk > > > > > of > > > > > changes required for HBase 3.0 support. (Mostly just getting rid of > > > HBase > > > > > 1.0 APIs) > > > > > I'd appreciate it if someone from your team could review them. See > > > > > PHOENIX-7164. I'd like to add support for HBase 3.0 > > > > > to Phoenix promptly after 3.0 is released. > > > > > > > > > > Istvan > > > > > > > > > > > > > > >> On Fri, Mar 7, 2025 at 2:56 AM Andrew Purtell <apurt...@apache.org> > > > wrote: > > > > >> > > > > >> I'm not sure what other user appetites for HBase 3 in production are, > > > but > > > > >> at the $dayjob it's got to be 18 months to 2 years out. First, we are > > > a > > > > >> Phoenix shop, so Phoenix must adapt their code for HBase 3 and > > > > >> prepare > > > > >> releases based on HBase 3 that will be production ready, and as far > > > as I > > > > >> know this isn't on the radar for them yet. (It's quite possible our > > > shop > > > > >> would do the work, but not until next year at the earliest.) Then, we > > > must > > > > >> update all of our internal code for API differences, and rigorously > > > test > > > > >> incremental upgrade and rollback, until we can move up and back > > > seamlessly, > > > > >> and that will take time. I think we might consider allocating > > > resources for > > > > >> it beginning next year. I don't think we are unusual in that regard. > > > > >> > > > > >> All this to say, maintenance changes like a Jetty uplift should not > > > > >> be > > > > >> restricted to 3.x only, because 2.x users are going to remain on 2.x > > > for a > > > > >> while, and 2.x users have just as much exposure to security blockers > > > > >> related to end of support versions of Jetty. If the project doesn't > > > land > > > > >> things like Jetty uplift back to 2.x, that's just more work for us to > > > do > > > > >> it, and less bandwidth to pursue more forward looking activities. > > > > >> > > > > >> For your consideration. > > > > >> > > > > >> On Thu, Mar 6, 2025 at 5:45 PM 张铎(Duo Zhang) <palomino...@gmail.com> > > > > >> wrote: > > > > >> > > > > >>> If we can speed up the 3.x release, since it has already moved up > > > > >>> the > > > > >>> JDK17+, it will be easier to land these changes in 3.x only. > > > > >>> > > > > >>> Istvan Toth <st...@cloudera.com.invalid> 于2025年3月7日周五 03:42写道： > > > > >>>> > > > > >>>> I'd split that into two tickets. > > > > >>>> > > > > >>>> First move to Jetty 12 while keeping the old javax package names, > > > and > > > > >>> once > > > > >>>> that's done and tested, we can look into moving to jakarta. > > > > >>>> > > > > >>>> Istvan > > > > >>>> > > > > >>>> On Thu, Mar 6, 2025 at 8:33 PM Andrew Purtell <apurt...@apache.org> > > > > >>> wrote: > > > > >>>> > > > > >>>>>> Therefore, I plan to use the EE8 environment on Jetty 12. > > > > >>>>> > > > > >>>>> Ah, thanks for mentioning EE. > > > > >>>>> > > > > >>>>> Do you have a requirement to stay with EE 8 ? > > > > >>>>> > > > > >>>>> If we could target EE 9 instead of EE 8 that would align with some > > > > >>> internal > > > > >>>>> requirements at $dayjob. For what it's worth. Mostly the point of > > > EE > > > > >> 9 > > > > >>> is a > > > > >>>>> move of the EE stuff into a new namespace, allowing for further > > > > >>> development > > > > >>>>> under the Eclipse Foundation without trademark conflicts with > > > > >> Oracle's > > > > >>>>> "Java" branding. The trademark issue and follow on Java licensing > > > > >>> concerns > > > > >>>>> are going to be relevant for anyone who might be visited by an > > > Oracle > > > > >>>>> lawyer-shark. > > > > >>>>> > > > > >>>>> On Tue, Mar 4, 2025 at 6:42 AM Nihal Jain <nihalj...@apache.org> > > > > >>> wrote: > > > > >>>>> > > > > >>>>>>> I think that the best course of action would be adding a > > > > >>>>>>> separate > > > > >>>>> Jetty11 > > > > >>>>>> module to hbase-thirdparty, this way 9.4.x and 11.0.x can be both > > > > >>>>>> used/updated by the 2.x / 3.x branches. > > > > >>>>>> > > > > >>>>>> +1, I would like to volunteer for this work. > > > > >>>>>> > > > > >>>>>> Maintaining Jetty 9.4.x could be quite challenging from a CVE > > > > >>> perspective > > > > >>>>>> since it has been EOCS (End of Community Support) since June > > > > >>>>>> 2022, > > > > >>> and > > > > >>>>> new > > > > >>>>>> releases may not be forthcoming for a long time. I faced similar > > > > >>> issues > > > > >>>>>> during our last hbase-thirdparty release. See this issue comment: > > > > >>>>>> > > > > >>>>> > > > > >>> > > > > >> > > > https://github.com/jetty/jetty.project/issues/12630#issuecomment-2604701092 > > > > >>>>>> > > > > >>>>>> Regarding the version and CVEs, I agree with @Andrew and suggest > > > > >>> that we > > > > >>>>>> jump directly to Jetty 12, bypassing Jetty 11, to support > > > > >>> javax.servlet > > > > >>>>> for > > > > >>>>>> JSP 2.3.1. Therefore, I plan to use the EE8 environment on Jetty > > > > >> 12. > > > > >>> See > > > > >>>>>> Jetty version history: > > > > >>> https://jetty.org/download.html#version-history > > > > >>>>>> > > > > >>>>>> @Istvan, is there any particular reason you recommend moving to > > > > >>> Jetty 11, > > > > >>>>>> which is also EOCS? > > > > >>>>>> > > > > >>>>>> If others are fine with me taking this up, I can create the > > > > >> necessary > > > > >>>>>> JIRAs for the Jetty migration project and start the work soon. > > > > >>>>>> > > > > >>>>>> Regards, > > > > >>>>>> Nihal > > > > >>>>>> > > > > >>>>>> On 2025/03/03 18:08:53 Wei-Chiu Chuang wrote: > > > > >>>>>>> FYI I think Hadoop will need to make similar moves too. > > > > >>>>>>> Sounds like Hadoop should start the work to drop JDK8 as well. I > > > > >>> know > > > > >>>>>> Steve > > > > >>>>>>> has a patch that requires JDK17+ due to Iceberg > > > > >>>>>>> https://github.com/apache/hadoop/pull/7316 > > > > >>>>>>> > > > > >>>>>>> ---------- Forwarded message --------- > > > > >>>>>>> From: Istvan Toth <st...@apache.org> > > > > >>>>>>> Date: Sun, Mar 2, 2025 at 9:41 PM > > > > >>>>>>> Subject: [DISCUSS] Plans for JDK22 / SecurityManager removal > > > > >>>>>>> To: HBase Dev List <dev@hbase.apache.org> > > > > >>>>>>> > > > > >>>>>>> > > > > >>>>>>> Hi! > > > > >>>>>>> > > > > >>>>>>> The last big incompatible JDK change is the JEP411/JEP486 > > > > >>>>> SecurityManager > > > > >>>>>>> removal process, which has gotten serious with JDK22, which > > > > >>> disables > > > > >>>>>>> Subject.doAs*() Subject.getSubject() by default (while providing > > > > >>> the > > > > >>>>> kind > > > > >>>>>>> of equivalent callAs() and current() methods), and kills > > > > >> automatic > > > > >>>>>> Subject > > > > >>>>>>> propagation to new threads. > > > > >>>>>>> > > > > >>>>>>> To add insult to injury the new methods are only available from > > > > >>> JDK18, > > > > >>>>> so > > > > >>>>>>> it is not possible to just move to the new API without breaking > > > > >>> JDK17. > > > > >>>>>>> > > > > >>>>>>> Some of this can only be solved in Hadoop (hopefully 3.5) , but > > > > >>> HBase > > > > >>>>> can > > > > >>>>>>> also take steps to move towards JDK22 compatibility in the > > > > >>> meantime. > > > > >>>>>>> > > > > >>>>>>> Here's the plan I have in mind: > > > > >>>>>>> > > > > >>>>>>> - Review and the dependencies WRT SecurityManage usage (just > > > > >>> search for > > > > >>>>>> API > > > > >>>>>>> calls) > > > > >>>>>>> - Update the dependencies as needed > > > > >>>>>>> > > > > >>>>>>> - Add a reflection-based compatibility Utility class based on > > > > >>> Jetty's > > > > >>>>>>> SecurityUtils, (or one of its derivatives) > > > > >>>>>>> - Replace the deprecated method calls with the compatibility > > > > >>> versions > > > > >>>>>>> - Wrap Runnables/Callables with Subject current()/callAs() > > > > >> methods. > > > > >>>>>>> > > > > >>>>>>> One large dependency I have identified is Jetty. > > > > >>>>>>> While technically we MAY be able to keep using 9.4 by overriding > > > > >>> its > > > > >>>>>>> default ThreadFactory, I think that considering that even > > > > >>> semi-formal > > > > >>>>>> Jetty > > > > >>>>>>> 9.4 support is on its very last legs (the latest update was > > > > >>>>>>> never > > > > >>> even > > > > >>>>>>> announced and is not listed in the downloads page), the > > > > >>> advantages of > > > > >>>>>>> updating to Jetty 11 on branch-3+ now far outweighs the added > > > > >>>>> maintenance > > > > >>>>>>> burden of the branch divergence. > > > > >>>>>>> > > > > >>>>>>> I think that the best course of action would be adding a > > > > >>>>>>> separate > > > > >>>>> Jetty11 > > > > >>>>>>> module to hbase-thirdparty, this way 9.4.x and 11.0.x can be > > > > >>>>>>> both > > > > >>>>>>> used/updated by the 2.x / 3.x branches. > > > > >>>>>>> > > > > >>>>>>> What do you think ? > > > > >>>>>>> > > > > >>>>>>> - Do you agree that JEP411/486 should be treated as a priority ? > > > > >>>>>>> - Would you volunteer for the dependency audit ? > > > > >>>>>>> - Do you agree with the Jetty update, and the plan outlined > > > > >> above ? > > > > >>>>>>> > > > > >>>>>>> Istvan > > > > >>>>>>> > > > > >>>>>>> ps: Sorry, no links because GMail seems to react very bady to > > > > >> any. > > > > >>>>> Please > > > > >>>>>>> use search. > > > > >>>>>>> > > > > >>>>>> > > > > >>>>>> > > > > >> --------------------------------------------------------------------- > > > > >>>>>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > > > > >>>>>> For additional commands, e-mail: > > > common-dev-h...@hadoop.apache.org > > > > >>>>>> > > > > >>>>>> > > > > >>>>> > > > > >>>>> -- > > > > >>>>> Best regards, > > > > >>>>> Andrew > > > > >>>>> > > > > >>>>> Unrest, ignorance distilled, nihilistic imbeciles - > > > > >>>>> It's what we’ve earned > > > > >>>>> Welcome, apocalypse, what’s taken you so long? > > > > >>>>> Bring us the fitting end that we’ve been counting on > > > > >>>>> - A23, Welcome, Apocalypse > > > > >>>>> > > > > >>>> > > > > >>>> > > > > >>>> -- > > > > >>>> *István Tóth* | Sr. Staff Software Engineer > > > > >>>> *Email*: st...@cloudera.com > > > > >>>> cloudera.com <https://www.cloudera.com> > > > > >>>> [image: Cloudera] <https://www.cloudera.com/> > > > > >>>> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > > > > >>>> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > > > > >>> Cloudera > > > > >>>> on LinkedIn] <https://www.linkedin.com/company/cloudera> > > > > >>>> ------------------------------ > > > > >>>> ------------------------------ > > > > >>> > > > > >> > > > > >> > > > > >> -- > > > > >> Best regards, > > > > >> Andrew > > > > >> > > > > >> Unrest, ignorance distilled, nihilistic imbeciles - > > > > >> It's what we’ve earned > > > > >> Welcome, apocalypse, what’s taken you so long? > > > > >> Bring us the fitting end that we’ve been counting on > > > > >> - A23, Welcome, Apocalypse > > > > >> > > > > > > > > > > > > > > > -- > > > > > *István Tóth* | Sr. Staff Software Engineer > > > > > *Email*: st...@cloudera.com > > > > > cloudera.com <https://www.cloudera.com> > > > > > [image: Cloudera] <https://www.cloudera.com/> > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > > > Cloudera > > > > > on LinkedIn] <https://www.linkedin.com/company/cloudera> > > > > > ------------------------------ > > > > > ------------------------------ > > > > > > > > > > > > > -- > > *István Tóth* | Sr. Staff Software Engineer > > *Email*: st...@cloudera.com > > cloudera.com <https://www.cloudera.com> > > [image: Cloudera] <https://www.cloudera.com/> > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera > > on LinkedIn] <https://www.linkedin.com/company/cloudera> > > ------------------------------ > > ------------------------------ > >
