So after HBASE-26773, we still have another problem that we can not use sun.nio.ch.DirectBuffer. The main usage is to get the address.
Let me provide a new util method in the hbase-unsafe module. Andrew Purtell <andrew.purt...@gmail.com> 于2022年2月26日周六 13:58写道： > That sounds good, looking forward to it. Please let me know if you want > help. > > > On Feb 25, 2022, at 9:56 PM, 张铎 <palomino...@gmail.com> wrote: > > > > Filed HBASE-26773 and tried, the result is not very good. Netty's > > PlatformDependent can not meet all the requirements. > > > > So now I prefer we have a new hbase-thirdparty-unsafe module to provide > the > > Unsafe ability without exposing sun.misc.Unsafe directly. > > > > Will give it a try. > > > > Thanks. > > > > Andrew Purtell <apurt...@apache.org> 于2022年2月24日周四 10:46写道： > > > >> Ok, a bridge in one place would minimize the risk. > >> > >>> On Wed, Feb 23, 2022 at 6:07 PM 张铎(Duo Zhang) <palomino...@gmail.com> > >>> wrote: > >>> > >>> I think this is a reasonable concern, we could do this on our own. > >>> > >>> But checking the commit history of PlatformDependent[1], the API is > >> pretty > >>> stable, so I think the risk of hitting a security issue but we can not > >>> upgrade due to API conflicts is very low. And if a security issue > >> requires > >>> big changes on PlatformDependent, then I assume we have to fix our own > >>> version as well... > >>> > >>> And we will not spread PlatformDependent references everywhere in our > >> code, > >>> we will still have a bridge, like the current UnsafeAccess. So if later > >> we > >>> decide to implement our own version, it will not introduce a big impact > >> to > >>> our code base. > >>> > >>> Thanks. > >>> > >>> > >>> [1] > >>> > >>> > >> > https://github.com/netty/netty/commits/4.1/common/src/main/java/io/netty/util/internal/PlatformDependent.java > >>> > >>> Andrew Purtell <apurt...@apache.org> 于2022年2月24日周四 01:12写道： > >>> > >>>> We often upgrade netty after there is a CVE announcement. This > >> produces a > >>>> pressure to upgrade in production. Use of an internal API is not a > good > >>>> practice because it can cause friction because of breaking changes > when > >>>> there is a need to upgrade all of a sudden. This is painful and > >> hopefully > >>>> we can exclude it as a possibility now when coming up with the plan. > We > >>> can > >>>> do the same thing they do in our own hbase-thirdparty without taking > >> the > >>>> dependency and achieve the same goal, right? > >>>> > >>>> On Mon, Feb 21, 2022 at 7:35 PM 张铎(Duo Zhang) <palomino...@gmail.com> > >>>> wrote: > >>>> > >>>>> For me, since we shade netty, there will be no conflict with other > >>> netty > >>>>> dependencies, so we can make sure the netty version we used is > >> exactly > >>>> the > >>>>> one we shaded in hbase-thirdparty. > >>>>> > >>>>> If later we want to upgrade netty version in hbase-thirdparty, and it > >>>>> breaks the api, then we can release a new hbase-thirdparty, and > >> upgrade > >>>> the > >>>>> main hbase repo to depend on the new hbase-thirdparty version, and > >> also > >>>>> modify the code in hbase accordingly. There will be no user visible > >>>>> affects. > >>>>> > >>>>> Anyway, let me have a try first. Even with netty's PaltformDependent > >>>> class, > >>>>> since it still references sun.misc.Unsafe, maybe we will still get an > >>>>> compile error... > >>>>> > >>>>> Thanks. > >>>>> > >>>>> Andrew Purtell <apurt...@apache.org> 于2022年2月22日周二 08:08写道： > >>>>> > >>>>>> Thanks for that. I understand better what you are proposing now. It > >>> is > >>>>>> clear that you have thought it through. > >>>>>> > >>>>>> So then the only question I have is this: Do you think it is a > >>> concern > >>>>> that > >>>>>> Netty's PlatformDependent class is in their > >> "io.netty.util.internal" > >>>>>> package? The 'internal' designation suggests to me they won't apply > >>> the > >>>>>> same care to not breaking users of it as they would for their > >> public > >>>>>> (non-"internal") API. I noticed this earlier but had other > >> questions > >>> at > >>>>>> that time. > >>>>>> > >>>>>> If it is a concern, we have the option of doing the same thing that > >>>> Netty > >>>>>> has done with their PlatformDependent class as a new utility class > >> of > >>>> our > >>>>>> own in hbase-thirdparty. This may impose special requirements on > >>>> building > >>>>>> hbase-thirdparty, or, at least, the module containing this wrapper, > >>> but > >>>>> it > >>>>>> is an option that would help us avoid external breaking changes. > >> For > >>>> your > >>>>>> consideration. > >>>>>> > >>>>>> On Mon, Feb 21, 2022 at 3:48 PM 张铎(Duo Zhang) < > >> palomino...@gmail.com > >>>> > >>>>>> wrote: > >>>>>> > >>>>>>> What I proposed here is for solving the problem… > >>>>>>> > >>>>>>> The problem is that when using —release 8, javac will not export > >>> the > >>>>>>> jdk.unsupported symbols, so using sun.misc.Unsafe will trigger a > >>>>> compile > >>>>>>> error. But at runtime if you export jdk.unsupported you can still > >>> use > >>>>>>> sun.misc.Unsafe. The related jdk issue said —release is only for > >>>> public > >>>>>> API > >>>>>>> so not exporting jdk.unsupported is the expected behavior. > >>>>>>> > >>>>>>> So what I proposed here is to remove direct dependency on > >>>>> sun.misc.Unsafe > >>>>>>> in HBase code, so at compile time there will be no problem. Betty > >>>> has a > >>>>>>> wrapper of Unsafe so I think we could have a try. At runtime if > >> we > >>>> have > >>>>>>> jdk.unsupported exported we could still actually make use of > >>>>>>> sun.misc.Unsafe. > >>>>>>> > >>>>>>> Feel free to correct me if I missed something or there are still > >>>>>> uncleared > >>>>>>> things. > >>>>>>> > >>>>>>> Thanks. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Andrew Purtell <andrew.purt...@gmail.com>于2022年2月22日 周二00:56写道： > >>>>>>> > >>>>>>>> As Nick discovered ‘—release’ doesn’t work for version 8 > >> anyway. > >>>>>>>> > >>>>>>>> > >>>>>>>>> On Feb 20, 2022, at 3:47 PM, 张铎 <palomino...@gmail.com> > >> wrote: > >>>>>>>>> > >>>>>>>>> I know, with —release 8 ByteBuffer will not be an problem > >> but > >>>> then > >>>>>>>>> sun.misc.Unsafe can not be used any more right? So the > >> problem > >>>> here > >>>>>> is > >>>>>>>> how > >>>>>>>>> to make use of Unsafe with —release 8. What I proposed here > >> is > >>>> that > >>>>>> we > >>>>>>>> just > >>>>>>>>> do not use it, then we can make use of —release 8 to address > >>> the > >>>>>>>> ByteBuffer > >>>>>>>>> problem. > >>>>>>>>> > >>>>>>>>> Thanks. > >>>>>>>>> > >>>>>>>>> Andrew Purtell <apurt...@apache.org>于2022年2月21日 周一03:32写道： > >>>>>>>>> > >>>>>>>>>> The problem that leads us to consider '--release 8', or what > >>> it > >>>>>> would > >>>>>>>>>> promise if it worked, isn't Unsafe, it is ByteBuffer. > >>>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > https://www.morling.dev/blog/bytebuffer-and-the-dreaded-nosuchmethoderror/ > >>>>>>>>>> . Although to your point Netty has ByteBuf... > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> On Sun, Feb 20, 2022 at 5:23 AM 张铎(Duo Zhang) < > >>>>>> palomino...@gmail.com > >>>>>>>> > >>>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> Since we have shaded netty, I think we could make use of > >>>>>>>>>>> netty's PlatformDependent[1] class to avoid referencing > >>> Unsafe > >>>>>>>> directly, > >>>>>>>>>>> then there will be no problem for us to use the '--release > >> 8' > >>>>>> option. > >>>>>>>>>>> > >>>>>>>>>>> If no big concerns, I will file an issue to remove all the > >>>>>>>>>> sun.misc.Unsafe > >>>>>>>>>>> references in our code base and make use of > >> PlatformDependent > >>>>> from > >>>>>>> the > >>>>>>>>>>> shaded netty. > >>>>>>>>>>> > >>>>>>>>>>> Thanks. > >>>>>>>>>>> > >>>>>>>>>>> 1. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > https://github.com/netty/netty/blob/4.1/common/src/main/java/io/netty/util/internal/PlatformDependent.java > >>>>>>>>>>> > >>>>>>>>>>> 张铎(Duo Zhang) <palomino...@gmail.com> 于2022年2月16日周三 > >> 14:12写道： > >>>>>>>>>>> > >>>>>>>>>>>> I think this could be done by some module tricks, where we > >>>>> build a > >>>>>>>>>>> special > >>>>>>>>>>>> module with -source 8 and -target 8.We have done similar > >>>> things > >>>>> in > >>>>>>> the > >>>>>>>>>>> past > >>>>>>>>>>>> to have hadoop1-compat and hadoop2-compact. > >>>>>>>>>>>> > >>>>>>>>>>>> Let me have a try. > >>>>>>>>>>>> > >>>>>>>>>>>> Sean Busbey <bus...@apache.org> 于2022年2月16日周三 13:31写道： > >>>>>>>>>>>> > >>>>>>>>>>>>> Unfortunately if we want to keep jdk8 working we can't > >> rely > >>>> on > >>>>>>>>>> building > >>>>>>>>>>>>> with the jdk release target option on newer jdks. > >>>>>>>>>>>>> > >>>>>>>>>>>>> We ran into this recently in HBASE-25465 where a JDK bug > >>> came > >>>>> up. > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Tue, Feb 15, 2022 at 8:12 PM 张铎(Duo Zhang) < > >>>>>>> palomino...@gmail.com > >>>>>>>>> > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> +1 on moving the minimum required java version for > >>> compiling > >>>>>> HBase > >>>>>>>>>> to > >>>>>>>>>>>>> Java > >>>>>>>>>>>>>> 11. But we could still use --release=8 to still support > >>> jdk8 > >>>>> at > >>>>>>>>>>> runtime. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Java 8 is still widely used, and I believe this will > >> last > >>>> for > >>>>>> even > >>>>>>>>>>> more > >>>>>>>>>>>>>> years, as lots of big companies such as RedHat, > >> Microsoft > >>>> and > >>>>>>> Amazon > >>>>>>>>>>> are > >>>>>>>>>>>>>> still shipping new jdk8 releases, I do not think it is > >>> time > >>>> to > >>>>>>> fully > >>>>>>>>>>>>> drop > >>>>>>>>>>>>>> the support of jdk8. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Thanks. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Sean Busbey <bus...@apache.org> 于2022年2月16日周三 09:57写道： > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> If we set the minJDK to 11 I believe that will > >>> effectively > >>>>>> remove > >>>>>>>>>>> jdk8 > >>>>>>>>>>>>>>> support, rather than "just" deprecate it. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> As we build out more robust testing I would be in favor > >>> of > >>>>>>> running > >>>>>>>>>>>>> less > >>>>>>>>>>>>>> of > >>>>>>>>>>>>>>> it on deprecated JDKs. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Tue, Feb 15, 2022, 16:10 Josh Elser < > >>> els...@apache.org> > >>>>>>> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Deprecating jdk8 for HBase 3 and requiring minJdk=11 > >>> seems > >>>>>>>>>>>>> reasonable > >>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>> me. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Gotta start pushing the issue somehow. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On 2/15/22 1:47 PM, Sean Busbey wrote: > >>>>>>>>>>>>>>>>> Hi folks! > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> It's been some time since we decided to stick to LTS > >>> JDK > >>>>>>>>>>> releases > >>>>>>>>>>>>> as > >>>>>>>>>>>>>> a > >>>>>>>>>>>>>>>> way > >>>>>>>>>>>>>>>>> of getting a handle on the JDK treadmill. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> What do folks think about deprecating JDK8? The > >>> openjdk8u > >>>>>>>>>>> project > >>>>>>>>>>>>> is > >>>>>>>>>>>>>>>> still > >>>>>>>>>>>>>>>>> going and there are commercial support options at > >> least > >>>>>>>>>> through > >>>>>>>>>>>>> 2030. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Deprecating it in HBase 3 would mean we could remove > >> it > >>>> in > >>>>>>>>>> HBase > >>>>>>>>>>>>> 4, > >>>>>>>>>>>>>> not > >>>>>>>>>>>>>>>>> that we would _have_ to remove it. The way I think > >>> about > >>>>>>>>>> likely > >>>>>>>>>>>>>> timing > >>>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>> these events goes like this: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> * HBase 2 started alphas in June 2017, betas in > >> January > >>>>> 2018, > >>>>>>>>>>> and > >>>>>>>>>>>>>> came > >>>>>>>>>>>>>>>> out > >>>>>>>>>>>>>>>>> in April 2018 > >>>>>>>>>>>>>>>>> * HBase 3 started alphas in July 2021, and as of Feb > >>> 2022 > >>>>> we > >>>>>>>>>>>>> haven't > >>>>>>>>>>>>>>>>> discussed how close we are to our stated beta goals > >>>>> (upgrades > >>>>>>>>>>> from > >>>>>>>>>>>>>>> active > >>>>>>>>>>>>>>>>> 2.x releases and removal of not-ready features). > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Given the above, in the absence of us specifically > >>>> pushing > >>>>> to > >>>>>>>>>>> roll > >>>>>>>>>>>>>>>> through > >>>>>>>>>>>>>>>>> major version numbers for some reason, I think a > >>>> reasonably > >>>>>>>>>>>>>>> conservative > >>>>>>>>>>>>>>>>> estimate is for HBase 3 to arrive in late 2022 or > >> early > >>>>> 2023 > >>>>>>>>>> and > >>>>>>>>>>>>> then > >>>>>>>>>>>>>>>> HBase > >>>>>>>>>>>>>>>>> 4 to start alphas in ~2025. An HBase 5 prior to 2030 > >>>> seems > >>>>>>>>>>>>> unlikely. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> That all said, our current reference guide section on > >>>> java > >>>>>>>>>>>>> versions > >>>>>>>>>>>>>>> does > >>>>>>>>>>>>>>>>> not sound very confident about JDK11 support. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> A Note on JDK11 * > >>>>>>>>>>>>>>>>>> Preliminary support for JDK11 is introduced with > >> HBase > >>>>>> 2.3.0. > >>>>>>>>>>>>> This > >>>>>>>>>>>>>>>>> support is limited to > >>>>>>>>>>>>>>>>>> compilation and running the full test suite. There > >> are > >>>>> open > >>>>>>>>>>>>>> questions > >>>>>>>>>>>>>>>>> regarding the runtime > >>>>>>>>>>>>>>>>>> compatibility of JDK11 with Apache ZooKeeper and > >>> Apache > >>>>>>>>>> Hadoop > >>>>>>>>>>>>>>>>> (HADOOP-15338). > >>>>>>>>>>>>>>>>>> Significantly, neither project has yet released a > >>>> version > >>>>>>>>>> with > >>>>>>>>>>>>>>> explicit > >>>>>>>>>>>>>>>>> runtime support for > >>>>>>>>>>>>>>>>>> JDK11. The remaining known issues in HBase are > >>>> catalogued > >>>>> in > >>>>>>>>>>>>>>>> HBASE-22972. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Since that blurb was written, Hadoop has added JDK11 > >>>>> support > >>>>>>>>>> [1] > >>>>>>>>>>>>> as > >>>>>>>>>>>>>> has > >>>>>>>>>>>>>>>>> ZooKeeper[2]. As a part of buttoning up our JDK11 > >>> support > >>>>> we > >>>>>>>>>>> could > >>>>>>>>>>>>>>> update > >>>>>>>>>>>>>>>>> our minimum supported versions of these projects to > >>> match > >>>>>> that > >>>>>>>>>>>>>> support. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> What do folks think? > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> [1]: > >> https://hadoop.apache.org/docs/r3.3.0/index.html > >>>>>>>>>>>>>>>>> [2]: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >> > https://zookeeper.apache.org/doc/r3.6.0/zookeeperAdmin.html#sc_systemReq > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>>> Best regards, > >>>>>>>>>> Andrew > >>>>>>>>>> > >>>>>>>>>> Unrest, ignorance distilled, nihilistic imbeciles - > >>>>>>>>>> It's what we’ve earned > >>>>>>>>>> Welcome, apocalypse, what’s taken you so long? > >>>>>>>>>> Bring us the fitting end that we’ve been counting on > >>>>>>>>>> - A23, Welcome, Apocalypse > >>>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>>> -- > >>>>>> Best regards, > >>>>>> Andrew > >>>>>> > >>>>>> Unrest, ignorance distilled, nihilistic imbeciles - > >>>>>> It's what we’ve earned > >>>>>> Welcome, apocalypse, what’s taken you so long? > >>>>>> Bring us the fitting end that we’ve been counting on > >>>>>> - A23, Welcome, Apocalypse > >>>>>> > >>>>> > >>>> > >>>> > >>>> -- > >>>> Best regards, > >>>> Andrew > >>>> > >>>> Unrest, ignorance distilled, nihilistic imbeciles - > >>>> It's what we’ve earned > >>>> Welcome, apocalypse, what’s taken you so long? > >>>> Bring us the fitting end that we’ve been counting on > >>>> - A23, Welcome, Apocalypse > >>>> > >>> > >> > >> > >> -- > >> Best regards, > >> Andrew > >> > >> Unrest, ignorance distilled, nihilistic imbeciles - > >> It's what we’ve earned > >> Welcome, apocalypse, what’s taken you so long? > >> Bring us the fitting end that we’ve been counting on > >> - A23, Welcome, Apocalypse > >> >
