That sounds good, looking forward to it. Please let me know if you want help.
> On Feb 25, 2022, at 9:56 PM, 张铎 <palomino...@gmail.com> wrote: > > Filed HBASE-26773 and tried, the result is not very good. Netty's > PlatformDependent can not meet all the requirements. > > So now I prefer we have a new hbase-thirdparty-unsafe module to provide the > Unsafe ability without exposing sun.misc.Unsafe directly. > > Will give it a try. > > Thanks. > > Andrew Purtell <apurt...@apache.org> 于2022年2月24日周四 10:46写道： > >> Ok, a bridge in one place would minimize the risk. >> >>> On Wed, Feb 23, 2022 at 6:07 PM 张铎(Duo Zhang) <palomino...@gmail.com> >>> wrote: >>> >>> I think this is a reasonable concern, we could do this on our own. >>> >>> But checking the commit history of PlatformDependent[1], the API is >> pretty >>> stable, so I think the risk of hitting a security issue but we can not >>> upgrade due to API conflicts is very low. And if a security issue >> requires >>> big changes on PlatformDependent, then I assume we have to fix our own >>> version as well... >>> >>> And we will not spread PlatformDependent references everywhere in our >> code, >>> we will still have a bridge, like the current UnsafeAccess. So if later >> we >>> decide to implement our own version, it will not introduce a big impact >> to >>> our code base. >>> >>> Thanks. >>> >>> >>> [1] >>> >>> >> https://github.com/netty/netty/commits/4.1/common/src/main/java/io/netty/util/internal/PlatformDependent.java >>> >>> Andrew Purtell <apurt...@apache.org> 于2022年2月24日周四 01:12写道： >>> >>>> We often upgrade netty after there is a CVE announcement. This >> produces a >>>> pressure to upgrade in production. Use of an internal API is not a good >>>> practice because it can cause friction because of breaking changes when >>>> there is a need to upgrade all of a sudden. This is painful and >> hopefully >>>> we can exclude it as a possibility now when coming up with the plan. We >>> can >>>> do the same thing they do in our own hbase-thirdparty without taking >> the >>>> dependency and achieve the same goal, right? >>>> >>>> On Mon, Feb 21, 2022 at 7:35 PM 张铎(Duo Zhang) <palomino...@gmail.com> >>>> wrote: >>>> >>>>> For me, since we shade netty, there will be no conflict with other >>> netty >>>>> dependencies, so we can make sure the netty version we used is >> exactly >>>> the >>>>> one we shaded in hbase-thirdparty. >>>>> >>>>> If later we want to upgrade netty version in hbase-thirdparty, and it >>>>> breaks the api, then we can release a new hbase-thirdparty, and >> upgrade >>>> the >>>>> main hbase repo to depend on the new hbase-thirdparty version, and >> also >>>>> modify the code in hbase accordingly. There will be no user visible >>>>> affects. >>>>> >>>>> Anyway, let me have a try first. Even with netty's PaltformDependent >>>> class, >>>>> since it still references sun.misc.Unsafe, maybe we will still get an >>>>> compile error... >>>>> >>>>> Thanks. >>>>> >>>>> Andrew Purtell <apurt...@apache.org> 于2022年2月22日周二 08:08写道： >>>>> >>>>>> Thanks for that. I understand better what you are proposing now. It >>> is >>>>>> clear that you have thought it through. >>>>>> >>>>>> So then the only question I have is this: Do you think it is a >>> concern >>>>> that >>>>>> Netty's PlatformDependent class is in their >> "io.netty.util.internal" >>>>>> package? The 'internal' designation suggests to me they won't apply >>> the >>>>>> same care to not breaking users of it as they would for their >> public >>>>>> (non-"internal") API. I noticed this earlier but had other >> questions >>> at >>>>>> that time. >>>>>> >>>>>> If it is a concern, we have the option of doing the same thing that >>>> Netty >>>>>> has done with their PlatformDependent class as a new utility class >> of >>>> our >>>>>> own in hbase-thirdparty. This may impose special requirements on >>>> building >>>>>> hbase-thirdparty, or, at least, the module containing this wrapper, >>> but >>>>> it >>>>>> is an option that would help us avoid external breaking changes. >> For >>>> your >>>>>> consideration. >>>>>> >>>>>> On Mon, Feb 21, 2022 at 3:48 PM 张铎(Duo Zhang) < >> palomino...@gmail.com >>>> >>>>>> wrote: >>>>>> >>>>>>> What I proposed here is for solving the problem… >>>>>>> >>>>>>> The problem is that when using —release 8, javac will not export >>> the >>>>>>> jdk.unsupported symbols, so using sun.misc.Unsafe will trigger a >>>>> compile >>>>>>> error. But at runtime if you export jdk.unsupported you can still >>> use >>>>>>> sun.misc.Unsafe. The related jdk issue said —release is only for >>>> public >>>>>> API >>>>>>> so not exporting jdk.unsupported is the expected behavior. >>>>>>> >>>>>>> So what I proposed here is to remove direct dependency on >>>>> sun.misc.Unsafe >>>>>>> in HBase code, so at compile time there will be no problem. Betty >>>> has a >>>>>>> wrapper of Unsafe so I think we could have a try. At runtime if >> we >>>> have >>>>>>> jdk.unsupported exported we could still actually make use of >>>>>>> sun.misc.Unsafe. >>>>>>> >>>>>>> Feel free to correct me if I missed something or there are still >>>>>> uncleared >>>>>>> things. >>>>>>> >>>>>>> Thanks. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Andrew Purtell <andrew.purt...@gmail.com>于2022年2月22日 周二00:56写道： >>>>>>> >>>>>>>> As Nick discovered ‘—release’ doesn’t work for version 8 >> anyway. >>>>>>>> >>>>>>>> >>>>>>>>> On Feb 20, 2022, at 3:47 PM, 张铎 <palomino...@gmail.com> >> wrote: >>>>>>>>> >>>>>>>>> I know, with —release 8 ByteBuffer will not be an problem >> but >>>> then >>>>>>>>> sun.misc.Unsafe can not be used any more right? So the >> problem >>>> here >>>>>> is >>>>>>>> how >>>>>>>>> to make use of Unsafe with —release 8. What I proposed here >> is >>>> that >>>>>> we >>>>>>>> just >>>>>>>>> do not use it, then we can make use of —release 8 to address >>> the >>>>>>>> ByteBuffer >>>>>>>>> problem. >>>>>>>>> >>>>>>>>> Thanks. >>>>>>>>> >>>>>>>>> Andrew Purtell <apurt...@apache.org>于2022年2月21日 周一03:32写道： >>>>>>>>> >>>>>>>>>> The problem that leads us to consider '--release 8', or what >>> it >>>>>> would >>>>>>>>>> promise if it worked, isn't Unsafe, it is ByteBuffer. >>>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> https://www.morling.dev/blog/bytebuffer-and-the-dreaded-nosuchmethoderror/ >>>>>>>>>> . Although to your point Netty has ByteBuf... >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> On Sun, Feb 20, 2022 at 5:23 AM 张铎(Duo Zhang) < >>>>>> palomino...@gmail.com >>>>>>>> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Since we have shaded netty, I think we could make use of >>>>>>>>>>> netty's PlatformDependent[1] class to avoid referencing >>> Unsafe >>>>>>>> directly, >>>>>>>>>>> then there will be no problem for us to use the '--release >> 8' >>>>>> option. >>>>>>>>>>> >>>>>>>>>>> If no big concerns, I will file an issue to remove all the >>>>>>>>>> sun.misc.Unsafe >>>>>>>>>>> references in our code base and make use of >> PlatformDependent >>>>> from >>>>>>> the >>>>>>>>>>> shaded netty. >>>>>>>>>>> >>>>>>>>>>> Thanks. >>>>>>>>>>> >>>>>>>>>>> 1. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> https://github.com/netty/netty/blob/4.1/common/src/main/java/io/netty/util/internal/PlatformDependent.java >>>>>>>>>>> >>>>>>>>>>> 张铎(Duo Zhang) <palomino...@gmail.com> 于2022年2月16日周三 >> 14:12写道： >>>>>>>>>>> >>>>>>>>>>>> I think this could be done by some module tricks, where we >>>>> build a >>>>>>>>>>> special >>>>>>>>>>>> module with -source 8 and -target 8.We have done similar >>>> things >>>>> in >>>>>>> the >>>>>>>>>>> past >>>>>>>>>>>> to have hadoop1-compat and hadoop2-compact. >>>>>>>>>>>> >>>>>>>>>>>> Let me have a try. >>>>>>>>>>>> >>>>>>>>>>>> Sean Busbey <bus...@apache.org> 于2022年2月16日周三 13:31写道： >>>>>>>>>>>> >>>>>>>>>>>>> Unfortunately if we want to keep jdk8 working we can't >> rely >>>> on >>>>>>>>>> building >>>>>>>>>>>>> with the jdk release target option on newer jdks. >>>>>>>>>>>>> >>>>>>>>>>>>> We ran into this recently in HBASE-25465 where a JDK bug >>> came >>>>> up. >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Feb 15, 2022 at 8:12 PM 张铎(Duo Zhang) < >>>>>>> palomino...@gmail.com >>>>>>>>> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> +1 on moving the minimum required java version for >>> compiling >>>>>> HBase >>>>>>>>>> to >>>>>>>>>>>>> Java >>>>>>>>>>>>>> 11. But we could still use --release=8 to still support >>> jdk8 >>>>> at >>>>>>>>>>> runtime. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Java 8 is still widely used, and I believe this will >> last >>>> for >>>>>> even >>>>>>>>>>> more >>>>>>>>>>>>>> years, as lots of big companies such as RedHat, >> Microsoft >>>> and >>>>>>> Amazon >>>>>>>>>>> are >>>>>>>>>>>>>> still shipping new jdk8 releases, I do not think it is >>> time >>>> to >>>>>>> fully >>>>>>>>>>>>> drop >>>>>>>>>>>>>> the support of jdk8. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Sean Busbey <bus...@apache.org> 于2022年2月16日周三 09:57写道： >>>>>>>>>>>>>> >>>>>>>>>>>>>>> If we set the minJDK to 11 I believe that will >>> effectively >>>>>> remove >>>>>>>>>>> jdk8 >>>>>>>>>>>>>>> support, rather than "just" deprecate it. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> As we build out more robust testing I would be in favor >>> of >>>>>>> running >>>>>>>>>>>>> less >>>>>>>>>>>>>> of >>>>>>>>>>>>>>> it on deprecated JDKs. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Tue, Feb 15, 2022, 16:10 Josh Elser < >>> els...@apache.org> >>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Deprecating jdk8 for HBase 3 and requiring minJdk=11 >>> seems >>>>>>>>>>>>> reasonable >>>>>>>>>>>>>> to >>>>>>>>>>>>>>>> me. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Gotta start pushing the issue somehow. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 2/15/22 1:47 PM, Sean Busbey wrote: >>>>>>>>>>>>>>>>> Hi folks! >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> It's been some time since we decided to stick to LTS >>> JDK >>>>>>>>>>> releases >>>>>>>>>>>>> as >>>>>>>>>>>>>> a >>>>>>>>>>>>>>>> way >>>>>>>>>>>>>>>>> of getting a handle on the JDK treadmill. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> What do folks think about deprecating JDK8? The >>> openjdk8u >>>>>>>>>>> project >>>>>>>>>>>>> is >>>>>>>>>>>>>>>> still >>>>>>>>>>>>>>>>> going and there are commercial support options at >> least >>>>>>>>>> through >>>>>>>>>>>>> 2030. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Deprecating it in HBase 3 would mean we could remove >> it >>>> in >>>>>>>>>> HBase >>>>>>>>>>>>> 4, >>>>>>>>>>>>>> not >>>>>>>>>>>>>>>>> that we would _have_ to remove it. The way I think >>> about >>>>>>>>>> likely >>>>>>>>>>>>>> timing >>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>> these events goes like this: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> * HBase 2 started alphas in June 2017, betas in >> January >>>>> 2018, >>>>>>>>>>> and >>>>>>>>>>>>>> came >>>>>>>>>>>>>>>> out >>>>>>>>>>>>>>>>> in April 2018 >>>>>>>>>>>>>>>>> * HBase 3 started alphas in July 2021, and as of Feb >>> 2022 >>>>> we >>>>>>>>>>>>> haven't >>>>>>>>>>>>>>>>> discussed how close we are to our stated beta goals >>>>> (upgrades >>>>>>>>>>> from >>>>>>>>>>>>>>> active >>>>>>>>>>>>>>>>> 2.x releases and removal of not-ready features). >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Given the above, in the absence of us specifically >>>> pushing >>>>> to >>>>>>>>>>> roll >>>>>>>>>>>>>>>> through >>>>>>>>>>>>>>>>> major version numbers for some reason, I think a >>>> reasonably >>>>>>>>>>>>>>> conservative >>>>>>>>>>>>>>>>> estimate is for HBase 3 to arrive in late 2022 or >> early >>>>> 2023 >>>>>>>>>> and >>>>>>>>>>>>> then >>>>>>>>>>>>>>>> HBase >>>>>>>>>>>>>>>>> 4 to start alphas in ~2025. An HBase 5 prior to 2030 >>>> seems >>>>>>>>>>>>> unlikely. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> That all said, our current reference guide section on >>>> java >>>>>>>>>>>>> versions >>>>>>>>>>>>>>> does >>>>>>>>>>>>>>>>> not sound very confident about JDK11 support. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> A Note on JDK11 * >>>>>>>>>>>>>>>>>> Preliminary support for JDK11 is introduced with >> HBase >>>>>> 2.3.0. >>>>>>>>>>>>> This >>>>>>>>>>>>>>>>> support is limited to >>>>>>>>>>>>>>>>>> compilation and running the full test suite. There >> are >>>>> open >>>>>>>>>>>>>> questions >>>>>>>>>>>>>>>>> regarding the runtime >>>>>>>>>>>>>>>>>> compatibility of JDK11 with Apache ZooKeeper and >>> Apache >>>>>>>>>> Hadoop >>>>>>>>>>>>>>>>> (HADOOP-15338). >>>>>>>>>>>>>>>>>> Significantly, neither project has yet released a >>>> version >>>>>>>>>> with >>>>>>>>>>>>>>> explicit >>>>>>>>>>>>>>>>> runtime support for >>>>>>>>>>>>>>>>>> JDK11. The remaining known issues in HBase are >>>> catalogued >>>>> in >>>>>>>>>>>>>>>> HBASE-22972. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Since that blurb was written, Hadoop has added JDK11 >>>>> support >>>>>>>>>> [1] >>>>>>>>>>>>> as >>>>>>>>>>>>>> has >>>>>>>>>>>>>>>>> ZooKeeper[2]. As a part of buttoning up our JDK11 >>> support >>>>> we >>>>>>>>>>> could >>>>>>>>>>>>>>> update >>>>>>>>>>>>>>>>> our minimum supported versions of these projects to >>> match >>>>>> that >>>>>>>>>>>>>> support. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> What do folks think? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [1]: >> https://hadoop.apache.org/docs/r3.3.0/index.html >>>>>>>>>>>>>>>>> [2]: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>> >>>>>> >>>> >> https://zookeeper.apache.org/doc/r3.6.0/zookeeperAdmin.html#sc_systemReq >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Best regards, >>>>>>>>>> Andrew >>>>>>>>>> >>>>>>>>>> Unrest, ignorance distilled, nihilistic imbeciles - >>>>>>>>>> It's what we’ve earned >>>>>>>>>> Welcome, apocalypse, what’s taken you so long? >>>>>>>>>> Bring us the fitting end that we’ve been counting on >>>>>>>>>> - A23, Welcome, Apocalypse >>>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Best regards, >>>>>> Andrew >>>>>> >>>>>> Unrest, ignorance distilled, nihilistic imbeciles - >>>>>> It's what we’ve earned >>>>>> Welcome, apocalypse, what’s taken you so long? >>>>>> Bring us the fitting end that we’ve been counting on >>>>>> - A23, Welcome, Apocalypse >>>>>> >>>>> >>>> >>>> >>>> -- >>>> Best regards, >>>> Andrew >>>> >>>> Unrest, ignorance distilled, nihilistic imbeciles - >>>> It's what we’ve earned >>>> Welcome, apocalypse, what’s taken you so long? >>>> Bring us the fitting end that we’ve been counting on >>>> - A23, Welcome, Apocalypse >>>> >>> >> >> >> -- >> Best regards, >> Andrew >> >> Unrest, ignorance distilled, nihilistic imbeciles - >> It's what we’ve earned >> Welcome, apocalypse, what’s taken you so long? >> Bring us the fitting end that we’ve been counting on >> - A23, Welcome, Apocalypse >>
