Good news, with HBASE-26781 in place[1], I successfully built hbase with java 11 --release 8. Of course I need to modify the hbase main code a bit, and also make use of netty's PlatformDependent for processing DirectByteBuffer.
1. https://github.com/apache/hbase-thirdparty/pull/79 张铎(Duo Zhang) <palomino...@gmail.com> 于2022年3月1日周二 22:07写道： > So after HBASE-26773, we still have another problem that we can not > use sun.nio.ch.DirectBuffer. The main usage is to get the address. > > Let me provide a new util method in the hbase-unsafe module. > > Andrew Purtell <andrew.purt...@gmail.com> 于2022年2月26日周六 13:58写道： > >> That sounds good, looking forward to it. Please let me know if you want >> help. >> >> > On Feb 25, 2022, at 9:56 PM, 张铎 <palomino...@gmail.com> wrote: >> > >> > Filed HBASE-26773 and tried, the result is not very good. Netty's >> > PlatformDependent can not meet all the requirements. >> > >> > So now I prefer we have a new hbase-thirdparty-unsafe module to provide >> the >> > Unsafe ability without exposing sun.misc.Unsafe directly. >> > >> > Will give it a try. >> > >> > Thanks. >> > >> > Andrew Purtell <apurt...@apache.org> 于2022年2月24日周四 10:46写道： >> > >> >> Ok, a bridge in one place would minimize the risk. >> >> >> >>> On Wed, Feb 23, 2022 at 6:07 PM 张铎(Duo Zhang) <palomino...@gmail.com> >> >>> wrote: >> >>> >> >>> I think this is a reasonable concern, we could do this on our own. >> >>> >> >>> But checking the commit history of PlatformDependent[1], the API is >> >> pretty >> >>> stable, so I think the risk of hitting a security issue but we can not >> >>> upgrade due to API conflicts is very low. And if a security issue >> >> requires >> >>> big changes on PlatformDependent, then I assume we have to fix our own >> >>> version as well... >> >>> >> >>> And we will not spread PlatformDependent references everywhere in our >> >> code, >> >>> we will still have a bridge, like the current UnsafeAccess. So if >> later >> >> we >> >>> decide to implement our own version, it will not introduce a big >> impact >> >> to >> >>> our code base. >> >>> >> >>> Thanks. >> >>> >> >>> >> >>> [1] >> >>> >> >>> >> >> >> https://github.com/netty/netty/commits/4.1/common/src/main/java/io/netty/util/internal/PlatformDependent.java >> >>> >> >>> Andrew Purtell <apurt...@apache.org> 于2022年2月24日周四 01:12写道： >> >>> >> >>>> We often upgrade netty after there is a CVE announcement. This >> >> produces a >> >>>> pressure to upgrade in production. Use of an internal API is not a >> good >> >>>> practice because it can cause friction because of breaking changes >> when >> >>>> there is a need to upgrade all of a sudden. This is painful and >> >> hopefully >> >>>> we can exclude it as a possibility now when coming up with the plan. >> We >> >>> can >> >>>> do the same thing they do in our own hbase-thirdparty without taking >> >> the >> >>>> dependency and achieve the same goal, right? >> >>>> >> >>>> On Mon, Feb 21, 2022 at 7:35 PM 张铎(Duo Zhang) <palomino...@gmail.com >> > >> >>>> wrote: >> >>>> >> >>>>> For me, since we shade netty, there will be no conflict with other >> >>> netty >> >>>>> dependencies, so we can make sure the netty version we used is >> >> exactly >> >>>> the >> >>>>> one we shaded in hbase-thirdparty. >> >>>>> >> >>>>> If later we want to upgrade netty version in hbase-thirdparty, and >> it >> >>>>> breaks the api, then we can release a new hbase-thirdparty, and >> >> upgrade >> >>>> the >> >>>>> main hbase repo to depend on the new hbase-thirdparty version, and >> >> also >> >>>>> modify the code in hbase accordingly. There will be no user visible >> >>>>> affects. >> >>>>> >> >>>>> Anyway, let me have a try first. Even with netty's PaltformDependent >> >>>> class, >> >>>>> since it still references sun.misc.Unsafe, maybe we will still get >> an >> >>>>> compile error... >> >>>>> >> >>>>> Thanks. >> >>>>> >> >>>>> Andrew Purtell <apurt...@apache.org> 于2022年2月22日周二 08:08写道： >> >>>>> >> >>>>>> Thanks for that. I understand better what you are proposing now. It >> >>> is >> >>>>>> clear that you have thought it through. >> >>>>>> >> >>>>>> So then the only question I have is this: Do you think it is a >> >>> concern >> >>>>> that >> >>>>>> Netty's PlatformDependent class is in their >> >> "io.netty.util.internal" >> >>>>>> package? The 'internal' designation suggests to me they won't apply >> >>> the >> >>>>>> same care to not breaking users of it as they would for their >> >> public >> >>>>>> (non-"internal") API. I noticed this earlier but had other >> >> questions >> >>> at >> >>>>>> that time. >> >>>>>> >> >>>>>> If it is a concern, we have the option of doing the same thing that >> >>>> Netty >> >>>>>> has done with their PlatformDependent class as a new utility class >> >> of >> >>>> our >> >>>>>> own in hbase-thirdparty. This may impose special requirements on >> >>>> building >> >>>>>> hbase-thirdparty, or, at least, the module containing this wrapper, >> >>> but >> >>>>> it >> >>>>>> is an option that would help us avoid external breaking changes. >> >> For >> >>>> your >> >>>>>> consideration. >> >>>>>> >> >>>>>> On Mon, Feb 21, 2022 at 3:48 PM 张铎(Duo Zhang) < >> >> palomino...@gmail.com >> >>>> >> >>>>>> wrote: >> >>>>>> >> >>>>>>> What I proposed here is for solving the problem… >> >>>>>>> >> >>>>>>> The problem is that when using —release 8, javac will not export >> >>> the >> >>>>>>> jdk.unsupported symbols, so using sun.misc.Unsafe will trigger a >> >>>>> compile >> >>>>>>> error. But at runtime if you export jdk.unsupported you can still >> >>> use >> >>>>>>> sun.misc.Unsafe. The related jdk issue said —release is only for >> >>>> public >> >>>>>> API >> >>>>>>> so not exporting jdk.unsupported is the expected behavior. >> >>>>>>> >> >>>>>>> So what I proposed here is to remove direct dependency on >> >>>>> sun.misc.Unsafe >> >>>>>>> in HBase code, so at compile time there will be no problem. Betty >> >>>> has a >> >>>>>>> wrapper of Unsafe so I think we could have a try. At runtime if >> >> we >> >>>> have >> >>>>>>> jdk.unsupported exported we could still actually make use of >> >>>>>>> sun.misc.Unsafe. >> >>>>>>> >> >>>>>>> Feel free to correct me if I missed something or there are still >> >>>>>> uncleared >> >>>>>>> things. >> >>>>>>> >> >>>>>>> Thanks. >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> Andrew Purtell <andrew.purt...@gmail.com>于2022年2月22日 周二00:56写道： >> >>>>>>> >> >>>>>>>> As Nick discovered ‘—release’ doesn’t work for version 8 >> >> anyway. >> >>>>>>>> >> >>>>>>>> >> >>>>>>>>> On Feb 20, 2022, at 3:47 PM, 张铎 <palomino...@gmail.com> >> >> wrote: >> >>>>>>>>> >> >>>>>>>>> I know, with —release 8 ByteBuffer will not be an problem >> >> but >> >>>> then >> >>>>>>>>> sun.misc.Unsafe can not be used any more right? So the >> >> problem >> >>>> here >> >>>>>> is >> >>>>>>>> how >> >>>>>>>>> to make use of Unsafe with —release 8. What I proposed here >> >> is >> >>>> that >> >>>>>> we >> >>>>>>>> just >> >>>>>>>>> do not use it, then we can make use of —release 8 to address >> >>> the >> >>>>>>>> ByteBuffer >> >>>>>>>>> problem. >> >>>>>>>>> >> >>>>>>>>> Thanks. >> >>>>>>>>> >> >>>>>>>>> Andrew Purtell <apurt...@apache.org>于2022年2月21日 周一03:32写道： >> >>>>>>>>> >> >>>>>>>>>> The problem that leads us to consider '--release 8', or what >> >>> it >> >>>>>> would >> >>>>>>>>>> promise if it worked, isn't Unsafe, it is ByteBuffer. >> >>>>>>>>>> >> >>>>>>>> >> >>>>>>> >> >>>>>> >> >>>>> >> >>>> >> >>> >> >> >> https://www.morling.dev/blog/bytebuffer-and-the-dreaded-nosuchmethoderror/ >> >>>>>>>>>> . Although to your point Netty has ByteBuf... >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>>> On Sun, Feb 20, 2022 at 5:23 AM 张铎(Duo Zhang) < >> >>>>>> palomino...@gmail.com >> >>>>>>>> >> >>>>>>>>>>> wrote: >> >>>>>>>>>>> >> >>>>>>>>>>> Since we have shaded netty, I think we could make use of >> >>>>>>>>>>> netty's PlatformDependent[1] class to avoid referencing >> >>> Unsafe >> >>>>>>>> directly, >> >>>>>>>>>>> then there will be no problem for us to use the '--release >> >> 8' >> >>>>>> option. >> >>>>>>>>>>> >> >>>>>>>>>>> If no big concerns, I will file an issue to remove all the >> >>>>>>>>>> sun.misc.Unsafe >> >>>>>>>>>>> references in our code base and make use of >> >> PlatformDependent >> >>>>> from >> >>>>>>> the >> >>>>>>>>>>> shaded netty. >> >>>>>>>>>>> >> >>>>>>>>>>> Thanks. >> >>>>>>>>>>> >> >>>>>>>>>>> 1. >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>> >> >>>>>>> >> >>>>>> >> >>>>> >> >>>> >> >>> >> >> >> https://github.com/netty/netty/blob/4.1/common/src/main/java/io/netty/util/internal/PlatformDependent.java >> >>>>>>>>>>> >> >>>>>>>>>>> 张铎(Duo Zhang) <palomino...@gmail.com> 于2022年2月16日周三 >> >> 14:12写道： >> >>>>>>>>>>> >> >>>>>>>>>>>> I think this could be done by some module tricks, where we >> >>>>> build a >> >>>>>>>>>>> special >> >>>>>>>>>>>> module with -source 8 and -target 8.We have done similar >> >>>> things >> >>>>> in >> >>>>>>> the >> >>>>>>>>>>> past >> >>>>>>>>>>>> to have hadoop1-compat and hadoop2-compact. >> >>>>>>>>>>>> >> >>>>>>>>>>>> Let me have a try. >> >>>>>>>>>>>> >> >>>>>>>>>>>> Sean Busbey <bus...@apache.org> 于2022年2月16日周三 13:31写道： >> >>>>>>>>>>>> >> >>>>>>>>>>>>> Unfortunately if we want to keep jdk8 working we can't >> >> rely >> >>>> on >> >>>>>>>>>> building >> >>>>>>>>>>>>> with the jdk release target option on newer jdks. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> We ran into this recently in HBASE-25465 where a JDK bug >> >>> came >> >>>>> up. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> On Tue, Feb 15, 2022 at 8:12 PM 张铎(Duo Zhang) < >> >>>>>>> palomino...@gmail.com >> >>>>>>>>> >> >>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>> >> >>>>>>>>>>>>>> +1 on moving the minimum required java version for >> >>> compiling >> >>>>>> HBase >> >>>>>>>>>> to >> >>>>>>>>>>>>> Java >> >>>>>>>>>>>>>> 11. But we could still use --release=8 to still support >> >>> jdk8 >> >>>>> at >> >>>>>>>>>>> runtime. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Java 8 is still widely used, and I believe this will >> >> last >> >>>> for >> >>>>>> even >> >>>>>>>>>>> more >> >>>>>>>>>>>>>> years, as lots of big companies such as RedHat, >> >> Microsoft >> >>>> and >> >>>>>>> Amazon >> >>>>>>>>>>> are >> >>>>>>>>>>>>>> still shipping new jdk8 releases, I do not think it is >> >>> time >> >>>> to >> >>>>>>> fully >> >>>>>>>>>>>>> drop >> >>>>>>>>>>>>>> the support of jdk8. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Thanks. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Sean Busbey <bus...@apache.org> 于2022年2月16日周三 09:57写道： >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> If we set the minJDK to 11 I believe that will >> >>> effectively >> >>>>>> remove >> >>>>>>>>>>> jdk8 >> >>>>>>>>>>>>>>> support, rather than "just" deprecate it. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> As we build out more robust testing I would be in favor >> >>> of >> >>>>>>> running >> >>>>>>>>>>>>> less >> >>>>>>>>>>>>>> of >> >>>>>>>>>>>>>>> it on deprecated JDKs. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> On Tue, Feb 15, 2022, 16:10 Josh Elser < >> >>> els...@apache.org> >> >>>>>>> wrote: >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> Deprecating jdk8 for HBase 3 and requiring minJdk=11 >> >>> seems >> >>>>>>>>>>>>> reasonable >> >>>>>>>>>>>>>> to >> >>>>>>>>>>>>>>>> me. >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> Gotta start pushing the issue somehow. >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> On 2/15/22 1:47 PM, Sean Busbey wrote: >> >>>>>>>>>>>>>>>>> Hi folks! >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> It's been some time since we decided to stick to LTS >> >>> JDK >> >>>>>>>>>>> releases >> >>>>>>>>>>>>> as >> >>>>>>>>>>>>>> a >> >>>>>>>>>>>>>>>> way >> >>>>>>>>>>>>>>>>> of getting a handle on the JDK treadmill. >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> What do folks think about deprecating JDK8? The >> >>> openjdk8u >> >>>>>>>>>>> project >> >>>>>>>>>>>>> is >> >>>>>>>>>>>>>>>> still >> >>>>>>>>>>>>>>>>> going and there are commercial support options at >> >> least >> >>>>>>>>>> through >> >>>>>>>>>>>>> 2030. >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> Deprecating it in HBase 3 would mean we could remove >> >> it >> >>>> in >> >>>>>>>>>> HBase >> >>>>>>>>>>>>> 4, >> >>>>>>>>>>>>>> not >> >>>>>>>>>>>>>>>>> that we would _have_ to remove it. The way I think >> >>> about >> >>>>>>>>>> likely >> >>>>>>>>>>>>>> timing >> >>>>>>>>>>>>>>> of >> >>>>>>>>>>>>>>>>> these events goes like this: >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> * HBase 2 started alphas in June 2017, betas in >> >> January >> >>>>> 2018, >> >>>>>>>>>>> and >> >>>>>>>>>>>>>> came >> >>>>>>>>>>>>>>>> out >> >>>>>>>>>>>>>>>>> in April 2018 >> >>>>>>>>>>>>>>>>> * HBase 3 started alphas in July 2021, and as of Feb >> >>> 2022 >> >>>>> we >> >>>>>>>>>>>>> haven't >> >>>>>>>>>>>>>>>>> discussed how close we are to our stated beta goals >> >>>>> (upgrades >> >>>>>>>>>>> from >> >>>>>>>>>>>>>>> active >> >>>>>>>>>>>>>>>>> 2.x releases and removal of not-ready features). >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> Given the above, in the absence of us specifically >> >>>> pushing >> >>>>> to >> >>>>>>>>>>> roll >> >>>>>>>>>>>>>>>> through >> >>>>>>>>>>>>>>>>> major version numbers for some reason, I think a >> >>>> reasonably >> >>>>>>>>>>>>>>> conservative >> >>>>>>>>>>>>>>>>> estimate is for HBase 3 to arrive in late 2022 or >> >> early >> >>>>> 2023 >> >>>>>>>>>> and >> >>>>>>>>>>>>> then >> >>>>>>>>>>>>>>>> HBase >> >>>>>>>>>>>>>>>>> 4 to start alphas in ~2025. An HBase 5 prior to 2030 >> >>>> seems >> >>>>>>>>>>>>> unlikely. >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> That all said, our current reference guide section on >> >>>> java >> >>>>>>>>>>>>> versions >> >>>>>>>>>>>>>>> does >> >>>>>>>>>>>>>>>>> not sound very confident about JDK11 support. >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> A Note on JDK11 * >> >>>>>>>>>>>>>>>>>> Preliminary support for JDK11 is introduced with >> >> HBase >> >>>>>> 2.3.0. >> >>>>>>>>>>>>> This >> >>>>>>>>>>>>>>>>> support is limited to >> >>>>>>>>>>>>>>>>>> compilation and running the full test suite. There >> >> are >> >>>>> open >> >>>>>>>>>>>>>> questions >> >>>>>>>>>>>>>>>>> regarding the runtime >> >>>>>>>>>>>>>>>>>> compatibility of JDK11 with Apache ZooKeeper and >> >>> Apache >> >>>>>>>>>> Hadoop >> >>>>>>>>>>>>>>>>> (HADOOP-15338). >> >>>>>>>>>>>>>>>>>> Significantly, neither project has yet released a >> >>>> version >> >>>>>>>>>> with >> >>>>>>>>>>>>>>> explicit >> >>>>>>>>>>>>>>>>> runtime support for >> >>>>>>>>>>>>>>>>>> JDK11. The remaining known issues in HBase are >> >>>> catalogued >> >>>>> in >> >>>>>>>>>>>>>>>> HBASE-22972. >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> Since that blurb was written, Hadoop has added JDK11 >> >>>>> support >> >>>>>>>>>> [1] >> >>>>>>>>>>>>> as >> >>>>>>>>>>>>>> has >> >>>>>>>>>>>>>>>>> ZooKeeper[2]. As a part of buttoning up our JDK11 >> >>> support >> >>>>> we >> >>>>>>>>>>> could >> >>>>>>>>>>>>>>> update >> >>>>>>>>>>>>>>>>> our minimum supported versions of these projects to >> >>> match >> >>>>>> that >> >>>>>>>>>>>>>> support. >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> What do folks think? >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> [1]: >> >> https://hadoop.apache.org/docs/r3.3.0/index.html >> >>>>>>>>>>>>>>>>> [2]: >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>> >> >>>>>> >> >>>> >> >> >> https://zookeeper.apache.org/doc/r3.6.0/zookeeperAdmin.html#sc_systemReq >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> -- >> >>>>>>>>>> Best regards, >> >>>>>>>>>> Andrew >> >>>>>>>>>> >> >>>>>>>>>> Unrest, ignorance distilled, nihilistic imbeciles - >> >>>>>>>>>> It's what we’ve earned >> >>>>>>>>>> Welcome, apocalypse, what’s taken you so long? >> >>>>>>>>>> Bring us the fitting end that we’ve been counting on >> >>>>>>>>>> - A23, Welcome, Apocalypse >> >>>>>>>>>> >> >>>>>>>> >> >>>>>>> >> >>>>>> >> >>>>>> >> >>>>>> -- >> >>>>>> Best regards, >> >>>>>> Andrew >> >>>>>> >> >>>>>> Unrest, ignorance distilled, nihilistic imbeciles - >> >>>>>> It's what we’ve earned >> >>>>>> Welcome, apocalypse, what’s taken you so long? >> >>>>>> Bring us the fitting end that we’ve been counting on >> >>>>>> - A23, Welcome, Apocalypse >> >>>>>> >> >>>>> >> >>>> >> >>>> >> >>>> -- >> >>>> Best regards, >> >>>> Andrew >> >>>> >> >>>> Unrest, ignorance distilled, nihilistic imbeciles - >> >>>> It's what we’ve earned >> >>>> Welcome, apocalypse, what’s taken you so long? >> >>>> Bring us the fitting end that we’ve been counting on >> >>>> - A23, Welcome, Apocalypse >> >>>> >> >>> >> >> >> >> >> >> -- >> >> Best regards, >> >> Andrew >> >> >> >> Unrest, ignorance distilled, nihilistic imbeciles - >> >> It's what we’ve earned >> >> Welcome, apocalypse, what’s taken you so long? >> >> Bring us the fitting end that we’ve been counting on >> >> - A23, Welcome, Apocalypse >> >> >> >
