Filed HBASE-26793 for making a new hbase-thirdparty release. Then will put the PR for HBASE-25465.
Thanks. 张铎(Duo Zhang) <[email protected]> 于2022年3月2日周三 11:19写道： > Good news, with HBASE-26781 in place[1], I successfully built hbase with > java 11 --release 8. Of course I need to modify the hbase main code a bit, > and also make use of netty's PlatformDependent for processing > DirectByteBuffer. > > 1. https://github.com/apache/hbase-thirdparty/pull/79 > > 张铎(Duo Zhang) <[email protected]> 于2022年3月1日周二 22:07写道： > >> So after HBASE-26773, we still have another problem that we can not >> use sun.nio.ch.DirectBuffer. The main usage is to get the address. >> >> Let me provide a new util method in the hbase-unsafe module. >> >> Andrew Purtell <[email protected]> 于2022年2月26日周六 13:58写道： >> >>> That sounds good, looking forward to it. Please let me know if you want >>> help. >>> >>> > On Feb 25, 2022, at 9:56 PM, 张铎 <[email protected]> wrote: >>> > >>> > Filed HBASE-26773 and tried, the result is not very good. Netty's >>> > PlatformDependent can not meet all the requirements. >>> > >>> > So now I prefer we have a new hbase-thirdparty-unsafe module to >>> provide the >>> > Unsafe ability without exposing sun.misc.Unsafe directly. >>> > >>> > Will give it a try. >>> > >>> > Thanks. >>> > >>> > Andrew Purtell <[email protected]> 于2022年2月24日周四 10:46写道： >>> > >>> >> Ok, a bridge in one place would minimize the risk. >>> >> >>> >>> On Wed, Feb 23, 2022 at 6:07 PM 张铎(Duo Zhang) <[email protected] >>> > >>> >>> wrote: >>> >>> >>> >>> I think this is a reasonable concern, we could do this on our own. >>> >>> >>> >>> But checking the commit history of PlatformDependent[1], the API is >>> >> pretty >>> >>> stable, so I think the risk of hitting a security issue but we can >>> not >>> >>> upgrade due to API conflicts is very low. And if a security issue >>> >> requires >>> >>> big changes on PlatformDependent, then I assume we have to fix our >>> own >>> >>> version as well... >>> >>> >>> >>> And we will not spread PlatformDependent references everywhere in our >>> >> code, >>> >>> we will still have a bridge, like the current UnsafeAccess. So if >>> later >>> >> we >>> >>> decide to implement our own version, it will not introduce a big >>> impact >>> >> to >>> >>> our code base. >>> >>> >>> >>> Thanks. >>> >>> >>> >>> >>> >>> [1] >>> >>> >>> >>> >>> >> >>> https://github.com/netty/netty/commits/4.1/common/src/main/java/io/netty/util/internal/PlatformDependent.java >>> >>> >>> >>> Andrew Purtell <[email protected]> 于2022年2月24日周四 01:12写道： >>> >>> >>> >>>> We often upgrade netty after there is a CVE announcement. This >>> >> produces a >>> >>>> pressure to upgrade in production. Use of an internal API is not a >>> good >>> >>>> practice because it can cause friction because of breaking changes >>> when >>> >>>> there is a need to upgrade all of a sudden. This is painful and >>> >> hopefully >>> >>>> we can exclude it as a possibility now when coming up with the >>> plan. We >>> >>> can >>> >>>> do the same thing they do in our own hbase-thirdparty without taking >>> >> the >>> >>>> dependency and achieve the same goal, right? >>> >>>> >>> >>>> On Mon, Feb 21, 2022 at 7:35 PM 张铎(Duo Zhang) < >>> [email protected]> >>> >>>> wrote: >>> >>>> >>> >>>>> For me, since we shade netty, there will be no conflict with other >>> >>> netty >>> >>>>> dependencies, so we can make sure the netty version we used is >>> >> exactly >>> >>>> the >>> >>>>> one we shaded in hbase-thirdparty. >>> >>>>> >>> >>>>> If later we want to upgrade netty version in hbase-thirdparty, and >>> it >>> >>>>> breaks the api, then we can release a new hbase-thirdparty, and >>> >> upgrade >>> >>>> the >>> >>>>> main hbase repo to depend on the new hbase-thirdparty version, and >>> >> also >>> >>>>> modify the code in hbase accordingly. There will be no user visible >>> >>>>> affects. >>> >>>>> >>> >>>>> Anyway, let me have a try first. Even with netty's >>> PaltformDependent >>> >>>> class, >>> >>>>> since it still references sun.misc.Unsafe, maybe we will still get >>> an >>> >>>>> compile error... >>> >>>>> >>> >>>>> Thanks. >>> >>>>> >>> >>>>> Andrew Purtell <[email protected]> 于2022年2月22日周二 08:08写道： >>> >>>>> >>> >>>>>> Thanks for that. I understand better what you are proposing now. >>> It >>> >>> is >>> >>>>>> clear that you have thought it through. >>> >>>>>> >>> >>>>>> So then the only question I have is this: Do you think it is a >>> >>> concern >>> >>>>> that >>> >>>>>> Netty's PlatformDependent class is in their >>> >> "io.netty.util.internal" >>> >>>>>> package? The 'internal' designation suggests to me they won't >>> apply >>> >>> the >>> >>>>>> same care to not breaking users of it as they would for their >>> >> public >>> >>>>>> (non-"internal") API. I noticed this earlier but had other >>> >> questions >>> >>> at >>> >>>>>> that time. >>> >>>>>> >>> >>>>>> If it is a concern, we have the option of doing the same thing >>> that >>> >>>> Netty >>> >>>>>> has done with their PlatformDependent class as a new utility class >>> >> of >>> >>>> our >>> >>>>>> own in hbase-thirdparty. This may impose special requirements on >>> >>>> building >>> >>>>>> hbase-thirdparty, or, at least, the module containing this >>> wrapper, >>> >>> but >>> >>>>> it >>> >>>>>> is an option that would help us avoid external breaking changes. >>> >> For >>> >>>> your >>> >>>>>> consideration. >>> >>>>>> >>> >>>>>> On Mon, Feb 21, 2022 at 3:48 PM 张铎(Duo Zhang) < >>> >> [email protected] >>> >>>> >>> >>>>>> wrote: >>> >>>>>> >>> >>>>>>> What I proposed here is for solving the problem… >>> >>>>>>> >>> >>>>>>> The problem is that when using —release 8, javac will not export >>> >>> the >>> >>>>>>> jdk.unsupported symbols, so using sun.misc.Unsafe will trigger a >>> >>>>> compile >>> >>>>>>> error. But at runtime if you export jdk.unsupported you can still >>> >>> use >>> >>>>>>> sun.misc.Unsafe. The related jdk issue said —release is only for >>> >>>> public >>> >>>>>> API >>> >>>>>>> so not exporting jdk.unsupported is the expected behavior. >>> >>>>>>> >>> >>>>>>> So what I proposed here is to remove direct dependency on >>> >>>>> sun.misc.Unsafe >>> >>>>>>> in HBase code, so at compile time there will be no problem. Betty >>> >>>> has a >>> >>>>>>> wrapper of Unsafe so I think we could have a try. At runtime if >>> >> we >>> >>>> have >>> >>>>>>> jdk.unsupported exported we could still actually make use of >>> >>>>>>> sun.misc.Unsafe. >>> >>>>>>> >>> >>>>>>> Feel free to correct me if I missed something or there are still >>> >>>>>> uncleared >>> >>>>>>> things. >>> >>>>>>> >>> >>>>>>> Thanks. >>> >>>>>>> >>> >>>>>>> >>> >>>>>>> >>> >>>>>>> Andrew Purtell <[email protected]>于2022年2月22日 周二00:56写道： >>> >>>>>>> >>> >>>>>>>> As Nick discovered ‘—release’ doesn’t work for version 8 >>> >> anyway. >>> >>>>>>>> >>> >>>>>>>> >>> >>>>>>>>> On Feb 20, 2022, at 3:47 PM, 张铎 <[email protected]> >>> >> wrote: >>> >>>>>>>>> >>> >>>>>>>>> I know, with —release 8 ByteBuffer will not be an problem >>> >> but >>> >>>> then >>> >>>>>>>>> sun.misc.Unsafe can not be used any more right? So the >>> >> problem >>> >>>> here >>> >>>>>> is >>> >>>>>>>> how >>> >>>>>>>>> to make use of Unsafe with —release 8. What I proposed here >>> >> is >>> >>>> that >>> >>>>>> we >>> >>>>>>>> just >>> >>>>>>>>> do not use it, then we can make use of —release 8 to address >>> >>> the >>> >>>>>>>> ByteBuffer >>> >>>>>>>>> problem. >>> >>>>>>>>> >>> >>>>>>>>> Thanks. >>> >>>>>>>>> >>> >>>>>>>>> Andrew Purtell <[email protected]>于2022年2月21日 周一03:32写道： >>> >>>>>>>>> >>> >>>>>>>>>> The problem that leads us to consider '--release 8', or what >>> >>> it >>> >>>>>> would >>> >>>>>>>>>> promise if it worked, isn't Unsafe, it is ByteBuffer. >>> >>>>>>>>>> >>> >>>>>>>> >>> >>>>>>> >>> >>>>>> >>> >>>>> >>> >>>> >>> >>> >>> >> >>> https://www.morling.dev/blog/bytebuffer-and-the-dreaded-nosuchmethoderror/ >>> >>>>>>>>>> . Although to your point Netty has ByteBuf... >>> >>>>>>>>>> >>> >>>>>>>>>> >>> >>>>>>>>>>> On Sun, Feb 20, 2022 at 5:23 AM 张铎(Duo Zhang) < >>> >>>>>> [email protected] >>> >>>>>>>> >>> >>>>>>>>>>> wrote: >>> >>>>>>>>>>> >>> >>>>>>>>>>> Since we have shaded netty, I think we could make use of >>> >>>>>>>>>>> netty's PlatformDependent[1] class to avoid referencing >>> >>> Unsafe >>> >>>>>>>> directly, >>> >>>>>>>>>>> then there will be no problem for us to use the '--release >>> >> 8' >>> >>>>>> option. >>> >>>>>>>>>>> >>> >>>>>>>>>>> If no big concerns, I will file an issue to remove all the >>> >>>>>>>>>> sun.misc.Unsafe >>> >>>>>>>>>>> references in our code base and make use of >>> >> PlatformDependent >>> >>>>> from >>> >>>>>>> the >>> >>>>>>>>>>> shaded netty. >>> >>>>>>>>>>> >>> >>>>>>>>>>> Thanks. >>> >>>>>>>>>>> >>> >>>>>>>>>>> 1. >>> >>>>>>>>>>> >>> >>>>>>>>>>> >>> >>>>>>>>>> >>> >>>>>>>> >>> >>>>>>> >>> >>>>>> >>> >>>>> >>> >>>> >>> >>> >>> >> >>> https://github.com/netty/netty/blob/4.1/common/src/main/java/io/netty/util/internal/PlatformDependent.java >>> >>>>>>>>>>> >>> >>>>>>>>>>> 张铎(Duo Zhang) <[email protected]> 于2022年2月16日周三 >>> >> 14:12写道： >>> >>>>>>>>>>> >>> >>>>>>>>>>>> I think this could be done by some module tricks, where we >>> >>>>> build a >>> >>>>>>>>>>> special >>> >>>>>>>>>>>> module with -source 8 and -target 8.We have done similar >>> >>>> things >>> >>>>> in >>> >>>>>>> the >>> >>>>>>>>>>> past >>> >>>>>>>>>>>> to have hadoop1-compat and hadoop2-compact. >>> >>>>>>>>>>>> >>> >>>>>>>>>>>> Let me have a try. >>> >>>>>>>>>>>> >>> >>>>>>>>>>>> Sean Busbey <[email protected]> 于2022年2月16日周三 13:31写道： >>> >>>>>>>>>>>> >>> >>>>>>>>>>>>> Unfortunately if we want to keep jdk8 working we can't >>> >> rely >>> >>>> on >>> >>>>>>>>>> building >>> >>>>>>>>>>>>> with the jdk release target option on newer jdks. >>> >>>>>>>>>>>>> >>> >>>>>>>>>>>>> We ran into this recently in HBASE-25465 where a JDK bug >>> >>> came >>> >>>>> up. >>> >>>>>>>>>>>>> >>> >>>>>>>>>>>>> On Tue, Feb 15, 2022 at 8:12 PM 张铎(Duo Zhang) < >>> >>>>>>> [email protected] >>> >>>>>>>>> >>> >>>>>>>>>>>>> wrote: >>> >>>>>>>>>>>>> >>> >>>>>>>>>>>>>> +1 on moving the minimum required java version for >>> >>> compiling >>> >>>>>> HBase >>> >>>>>>>>>> to >>> >>>>>>>>>>>>> Java >>> >>>>>>>>>>>>>> 11. But we could still use --release=8 to still support >>> >>> jdk8 >>> >>>>> at >>> >>>>>>>>>>> runtime. >>> >>>>>>>>>>>>>> >>> >>>>>>>>>>>>>> Java 8 is still widely used, and I believe this will >>> >> last >>> >>>> for >>> >>>>>> even >>> >>>>>>>>>>> more >>> >>>>>>>>>>>>>> years, as lots of big companies such as RedHat, >>> >> Microsoft >>> >>>> and >>> >>>>>>> Amazon >>> >>>>>>>>>>> are >>> >>>>>>>>>>>>>> still shipping new jdk8 releases, I do not think it is >>> >>> time >>> >>>> to >>> >>>>>>> fully >>> >>>>>>>>>>>>> drop >>> >>>>>>>>>>>>>> the support of jdk8. >>> >>>>>>>>>>>>>> >>> >>>>>>>>>>>>>> Thanks. >>> >>>>>>>>>>>>>> >>> >>>>>>>>>>>>>> Sean Busbey <[email protected]> 于2022年2月16日周三 09:57写道： >>> >>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>> If we set the minJDK to 11 I believe that will >>> >>> effectively >>> >>>>>> remove >>> >>>>>>>>>>> jdk8 >>> >>>>>>>>>>>>>>> support, rather than "just" deprecate it. >>> >>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>> As we build out more robust testing I would be in favor >>> >>> of >>> >>>>>>> running >>> >>>>>>>>>>>>> less >>> >>>>>>>>>>>>>> of >>> >>>>>>>>>>>>>>> it on deprecated JDKs. >>> >>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>> On Tue, Feb 15, 2022, 16:10 Josh Elser < >>> >>> [email protected]> >>> >>>>>>> wrote: >>> >>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>> Deprecating jdk8 for HBase 3 and requiring minJdk=11 >>> >>> seems >>> >>>>>>>>>>>>> reasonable >>> >>>>>>>>>>>>>> to >>> >>>>>>>>>>>>>>>> me. >>> >>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>> Gotta start pushing the issue somehow. >>> >>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>> On 2/15/22 1:47 PM, Sean Busbey wrote: >>> >>>>>>>>>>>>>>>>> Hi folks! >>> >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> It's been some time since we decided to stick to LTS >>> >>> JDK >>> >>>>>>>>>>> releases >>> >>>>>>>>>>>>> as >>> >>>>>>>>>>>>>> a >>> >>>>>>>>>>>>>>>> way >>> >>>>>>>>>>>>>>>>> of getting a handle on the JDK treadmill. >>> >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> What do folks think about deprecating JDK8? The >>> >>> openjdk8u >>> >>>>>>>>>>> project >>> >>>>>>>>>>>>> is >>> >>>>>>>>>>>>>>>> still >>> >>>>>>>>>>>>>>>>> going and there are commercial support options at >>> >> least >>> >>>>>>>>>> through >>> >>>>>>>>>>>>> 2030. >>> >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> Deprecating it in HBase 3 would mean we could remove >>> >> it >>> >>>> in >>> >>>>>>>>>> HBase >>> >>>>>>>>>>>>> 4, >>> >>>>>>>>>>>>>> not >>> >>>>>>>>>>>>>>>>> that we would _have_ to remove it. The way I think >>> >>> about >>> >>>>>>>>>> likely >>> >>>>>>>>>>>>>> timing >>> >>>>>>>>>>>>>>> of >>> >>>>>>>>>>>>>>>>> these events goes like this: >>> >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> * HBase 2 started alphas in June 2017, betas in >>> >> January >>> >>>>> 2018, >>> >>>>>>>>>>> and >>> >>>>>>>>>>>>>> came >>> >>>>>>>>>>>>>>>> out >>> >>>>>>>>>>>>>>>>> in April 2018 >>> >>>>>>>>>>>>>>>>> * HBase 3 started alphas in July 2021, and as of Feb >>> >>> 2022 >>> >>>>> we >>> >>>>>>>>>>>>> haven't >>> >>>>>>>>>>>>>>>>> discussed how close we are to our stated beta goals >>> >>>>> (upgrades >>> >>>>>>>>>>> from >>> >>>>>>>>>>>>>>> active >>> >>>>>>>>>>>>>>>>> 2.x releases and removal of not-ready features). >>> >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> Given the above, in the absence of us specifically >>> >>>> pushing >>> >>>>> to >>> >>>>>>>>>>> roll >>> >>>>>>>>>>>>>>>> through >>> >>>>>>>>>>>>>>>>> major version numbers for some reason, I think a >>> >>>> reasonably >>> >>>>>>>>>>>>>>> conservative >>> >>>>>>>>>>>>>>>>> estimate is for HBase 3 to arrive in late 2022 or >>> >> early >>> >>>>> 2023 >>> >>>>>>>>>> and >>> >>>>>>>>>>>>> then >>> >>>>>>>>>>>>>>>> HBase >>> >>>>>>>>>>>>>>>>> 4 to start alphas in ~2025. An HBase 5 prior to 2030 >>> >>>> seems >>> >>>>>>>>>>>>> unlikely. >>> >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> That all said, our current reference guide section on >>> >>>> java >>> >>>>>>>>>>>>> versions >>> >>>>>>>>>>>>>>> does >>> >>>>>>>>>>>>>>>>> not sound very confident about JDK11 support. >>> >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>> A Note on JDK11 * >>> >>>>>>>>>>>>>>>>>> Preliminary support for JDK11 is introduced with >>> >> HBase >>> >>>>>> 2.3.0. >>> >>>>>>>>>>>>> This >>> >>>>>>>>>>>>>>>>> support is limited to >>> >>>>>>>>>>>>>>>>>> compilation and running the full test suite. There >>> >> are >>> >>>>> open >>> >>>>>>>>>>>>>> questions >>> >>>>>>>>>>>>>>>>> regarding the runtime >>> >>>>>>>>>>>>>>>>>> compatibility of JDK11 with Apache ZooKeeper and >>> >>> Apache >>> >>>>>>>>>> Hadoop >>> >>>>>>>>>>>>>>>>> (HADOOP-15338). >>> >>>>>>>>>>>>>>>>>> Significantly, neither project has yet released a >>> >>>> version >>> >>>>>>>>>> with >>> >>>>>>>>>>>>>>> explicit >>> >>>>>>>>>>>>>>>>> runtime support for >>> >>>>>>>>>>>>>>>>>> JDK11. The remaining known issues in HBase are >>> >>>> catalogued >>> >>>>> in >>> >>>>>>>>>>>>>>>> HBASE-22972. >>> >>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> Since that blurb was written, Hadoop has added JDK11 >>> >>>>> support >>> >>>>>>>>>> [1] >>> >>>>>>>>>>>>> as >>> >>>>>>>>>>>>>> has >>> >>>>>>>>>>>>>>>>> ZooKeeper[2]. As a part of buttoning up our JDK11 >>> >>> support >>> >>>>> we >>> >>>>>>>>>>> could >>> >>>>>>>>>>>>>>> update >>> >>>>>>>>>>>>>>>>> our minimum supported versions of these projects to >>> >>> match >>> >>>>>> that >>> >>>>>>>>>>>>>> support. >>> >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> What do folks think? >>> >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> [1]: >>> >> https://hadoop.apache.org/docs/r3.3.0/index.html >>> >>>>>>>>>>>>>>>>> [2]: >>> >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>> >>> >>>>>>>>>>>>> >>> >>>>>>>>>>> >>> >>>>>>>> >>> >>>>>> >>> >>>> >>> >> >>> https://zookeeper.apache.org/doc/r3.6.0/zookeeperAdmin.html#sc_systemReq >>> >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>> >>> >>>>>>>>>>>>> >>> >>>>>>>>>>>> >>> >>>>>>>>>>> >>> >>>>>>>>>> >>> >>>>>>>>>> >>> >>>>>>>>>> -- >>> >>>>>>>>>> Best regards, >>> >>>>>>>>>> Andrew >>> >>>>>>>>>> >>> >>>>>>>>>> Unrest, ignorance distilled, nihilistic imbeciles - >>> >>>>>>>>>> It's what we’ve earned >>> >>>>>>>>>> Welcome, apocalypse, what’s taken you so long? >>> >>>>>>>>>> Bring us the fitting end that we’ve been counting on >>> >>>>>>>>>> - A23, Welcome, Apocalypse >>> >>>>>>>>>> >>> >>>>>>>> >>> >>>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> -- >>> >>>>>> Best regards, >>> >>>>>> Andrew >>> >>>>>> >>> >>>>>> Unrest, ignorance distilled, nihilistic imbeciles - >>> >>>>>> It's what we’ve earned >>> >>>>>> Welcome, apocalypse, what’s taken you so long? >>> >>>>>> Bring us the fitting end that we’ve been counting on >>> >>>>>> - A23, Welcome, Apocalypse >>> >>>>>> >>> >>>>> >>> >>>> >>> >>>> >>> >>>> -- >>> >>>> Best regards, >>> >>>> Andrew >>> >>>> >>> >>>> Unrest, ignorance distilled, nihilistic imbeciles - >>> >>>> It's what we’ve earned >>> >>>> Welcome, apocalypse, what’s taken you so long? >>> >>>> Bring us the fitting end that we’ve been counting on >>> >>>> - A23, Welcome, Apocalypse >>> >>>> >>> >>> >>> >> >>> >> >>> >> -- >>> >> Best regards, >>> >> Andrew >>> >> >>> >> Unrest, ignorance distilled, nihilistic imbeciles - >>> >> It's what we’ve earned >>> >> Welcome, apocalypse, what’s taken you so long? >>> >> Bring us the fitting end that we’ve been counting on >>> >> - A23, Welcome, Apocalypse >>> >> >>> >>
