Filed HBASE-26773 and tried, the result is not very good. Netty's PlatformDependent can not meet all the requirements.
So now I prefer we have a new hbase-thirdparty-unsafe module to provide the Unsafe ability without exposing sun.misc.Unsafe directly. Will give it a try. Thanks. Andrew Purtell <apurt...@apache.org> 于2022年2月24日周四 10:46写道： > Ok, a bridge in one place would minimize the risk. > > On Wed, Feb 23, 2022 at 6:07 PM 张铎(Duo Zhang) <palomino...@gmail.com> > wrote: > > > I think this is a reasonable concern, we could do this on our own. > > > > But checking the commit history of PlatformDependent[1], the API is > pretty > > stable, so I think the risk of hitting a security issue but we can not > > upgrade due to API conflicts is very low. And if a security issue > requires > > big changes on PlatformDependent, then I assume we have to fix our own > > version as well... > > > > And we will not spread PlatformDependent references everywhere in our > code, > > we will still have a bridge, like the current UnsafeAccess. So if later > we > > decide to implement our own version, it will not introduce a big impact > to > > our code base. > > > > Thanks. > > > > > > [1] > > > > > https://github.com/netty/netty/commits/4.1/common/src/main/java/io/netty/util/internal/PlatformDependent.java > > > > Andrew Purtell <apurt...@apache.org> 于2022年2月24日周四 01:12写道： > > > > > We often upgrade netty after there is a CVE announcement. This > produces a > > > pressure to upgrade in production. Use of an internal API is not a good > > > practice because it can cause friction because of breaking changes when > > > there is a need to upgrade all of a sudden. This is painful and > hopefully > > > we can exclude it as a possibility now when coming up with the plan. We > > can > > > do the same thing they do in our own hbase-thirdparty without taking > the > > > dependency and achieve the same goal, right? > > > > > > On Mon, Feb 21, 2022 at 7:35 PM 张铎(Duo Zhang) <palomino...@gmail.com> > > > wrote: > > > > > > > For me, since we shade netty, there will be no conflict with other > > netty > > > > dependencies, so we can make sure the netty version we used is > exactly > > > the > > > > one we shaded in hbase-thirdparty. > > > > > > > > If later we want to upgrade netty version in hbase-thirdparty, and it > > > > breaks the api, then we can release a new hbase-thirdparty, and > upgrade > > > the > > > > main hbase repo to depend on the new hbase-thirdparty version, and > also > > > > modify the code in hbase accordingly. There will be no user visible > > > > affects. > > > > > > > > Anyway, let me have a try first. Even with netty's PaltformDependent > > > class, > > > > since it still references sun.misc.Unsafe, maybe we will still get an > > > > compile error... > > > > > > > > Thanks. > > > > > > > > Andrew Purtell <apurt...@apache.org> 于2022年2月22日周二 08:08写道： > > > > > > > > > Thanks for that. I understand better what you are proposing now. It > > is > > > > > clear that you have thought it through. > > > > > > > > > > So then the only question I have is this: Do you think it is a > > concern > > > > that > > > > > Netty's PlatformDependent class is in their > "io.netty.util.internal" > > > > > package? The 'internal' designation suggests to me they won't apply > > the > > > > > same care to not breaking users of it as they would for their > public > > > > > (non-"internal") API. I noticed this earlier but had other > questions > > at > > > > > that time. > > > > > > > > > > If it is a concern, we have the option of doing the same thing that > > > Netty > > > > > has done with their PlatformDependent class as a new utility class > of > > > our > > > > > own in hbase-thirdparty. This may impose special requirements on > > > building > > > > > hbase-thirdparty, or, at least, the module containing this wrapper, > > but > > > > it > > > > > is an option that would help us avoid external breaking changes. > For > > > your > > > > > consideration. > > > > > > > > > > On Mon, Feb 21, 2022 at 3:48 PM 张铎(Duo Zhang) < > palomino...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > What I proposed here is for solving the problem… > > > > > > > > > > > > The problem is that when using —release 8, javac will not export > > the > > > > > > jdk.unsupported symbols, so using sun.misc.Unsafe will trigger a > > > > compile > > > > > > error. But at runtime if you export jdk.unsupported you can still > > use > > > > > > sun.misc.Unsafe. The related jdk issue said —release is only for > > > public > > > > > API > > > > > > so not exporting jdk.unsupported is the expected behavior. > > > > > > > > > > > > So what I proposed here is to remove direct dependency on > > > > sun.misc.Unsafe > > > > > > in HBase code, so at compile time there will be no problem. Betty > > > has a > > > > > > wrapper of Unsafe so I think we could have a try. At runtime if > we > > > have > > > > > > jdk.unsupported exported we could still actually make use of > > > > > > sun.misc.Unsafe. > > > > > > > > > > > > Feel free to correct me if I missed something or there are still > > > > > uncleared > > > > > > things. > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > > > > > > > > > Andrew Purtell <andrew.purt...@gmail.com>于2022年2月22日 周二00:56写道： > > > > > > > > > > > > > As Nick discovered ‘—release’ doesn’t work for version 8 > anyway. > > > > > > > > > > > > > > > > > > > > > > On Feb 20, 2022, at 3:47 PM, 张铎 <palomino...@gmail.com> > wrote: > > > > > > > > > > > > > > > > I know, with —release 8 ByteBuffer will not be an problem > but > > > then > > > > > > > > sun.misc.Unsafe can not be used any more right? So the > problem > > > here > > > > > is > > > > > > > how > > > > > > > > to make use of Unsafe with —release 8. What I proposed here > is > > > that > > > > > we > > > > > > > just > > > > > > > > do not use it, then we can make use of —release 8 to address > > the > > > > > > > ByteBuffer > > > > > > > > problem. > > > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > Andrew Purtell <apurt...@apache.org>于2022年2月21日 周一03:32写道： > > > > > > > > > > > > > > > >> The problem that leads us to consider '--release 8', or what > > it > > > > > would > > > > > > > >> promise if it worked, isn't Unsafe, it is ByteBuffer. > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://www.morling.dev/blog/bytebuffer-and-the-dreaded-nosuchmethoderror/ > > > > > > > >> . Although to your point Netty has ByteBuf... > > > > > > > >> > > > > > > > >> > > > > > > > >>> On Sun, Feb 20, 2022 at 5:23 AM 张铎(Duo Zhang) < > > > > > palomino...@gmail.com > > > > > > > > > > > > > > >>> wrote: > > > > > > > >>> > > > > > > > >>> Since we have shaded netty, I think we could make use of > > > > > > > >>> netty's PlatformDependent[1] class to avoid referencing > > Unsafe > > > > > > > directly, > > > > > > > >>> then there will be no problem for us to use the '--release > 8' > > > > > option. > > > > > > > >>> > > > > > > > >>> If no big concerns, I will file an issue to remove all the > > > > > > > >> sun.misc.Unsafe > > > > > > > >>> references in our code base and make use of > PlatformDependent > > > > from > > > > > > the > > > > > > > >>> shaded netty. > > > > > > > >>> > > > > > > > >>> Thanks. > > > > > > > >>> > > > > > > > >>> 1. > > > > > > > >>> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/netty/netty/blob/4.1/common/src/main/java/io/netty/util/internal/PlatformDependent.java > > > > > > > >>> > > > > > > > >>> 张铎(Duo Zhang) <palomino...@gmail.com> 于2022年2月16日周三 > 14:12写道： > > > > > > > >>> > > > > > > > >>>> I think this could be done by some module tricks, where we > > > > build a > > > > > > > >>> special > > > > > > > >>>> module with -source 8 and -target 8.We have done similar > > > things > > > > in > > > > > > the > > > > > > > >>> past > > > > > > > >>>> to have hadoop1-compat and hadoop2-compact. > > > > > > > >>>> > > > > > > > >>>> Let me have a try. > > > > > > > >>>> > > > > > > > >>>> Sean Busbey <bus...@apache.org> 于2022年2月16日周三 13:31写道： > > > > > > > >>>> > > > > > > > >>>>> Unfortunately if we want to keep jdk8 working we can't > rely > > > on > > > > > > > >> building > > > > > > > >>>>> with the jdk release target option on newer jdks. > > > > > > > >>>>> > > > > > > > >>>>> We ran into this recently in HBASE-25465 where a JDK bug > > came > > > > up. > > > > > > > >>>>> > > > > > > > >>>>> On Tue, Feb 15, 2022 at 8:12 PM 张铎(Duo Zhang) < > > > > > > palomino...@gmail.com > > > > > > > > > > > > > > > >>>>> wrote: > > > > > > > >>>>> > > > > > > > >>>>>> +1 on moving the minimum required java version for > > compiling > > > > > HBase > > > > > > > >> to > > > > > > > >>>>> Java > > > > > > > >>>>>> 11. But we could still use --release=8 to still support > > jdk8 > > > > at > > > > > > > >>> runtime. > > > > > > > >>>>>> > > > > > > > >>>>>> Java 8 is still widely used, and I believe this will > last > > > for > > > > > even > > > > > > > >>> more > > > > > > > >>>>>> years, as lots of big companies such as RedHat, > Microsoft > > > and > > > > > > Amazon > > > > > > > >>> are > > > > > > > >>>>>> still shipping new jdk8 releases, I do not think it is > > time > > > to > > > > > > fully > > > > > > > >>>>> drop > > > > > > > >>>>>> the support of jdk8. > > > > > > > >>>>>> > > > > > > > >>>>>> Thanks. > > > > > > > >>>>>> > > > > > > > >>>>>> Sean Busbey <bus...@apache.org> 于2022年2月16日周三 09:57写道： > > > > > > > >>>>>> > > > > > > > >>>>>>> If we set the minJDK to 11 I believe that will > > effectively > > > > > remove > > > > > > > >>> jdk8 > > > > > > > >>>>>>> support, rather than "just" deprecate it. > > > > > > > >>>>>>> > > > > > > > >>>>>>> As we build out more robust testing I would be in favor > > of > > > > > > running > > > > > > > >>>>> less > > > > > > > >>>>>> of > > > > > > > >>>>>>> it on deprecated JDKs. > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> On Tue, Feb 15, 2022, 16:10 Josh Elser < > > els...@apache.org> > > > > > > wrote: > > > > > > > >>>>>>> > > > > > > > >>>>>>>> Deprecating jdk8 for HBase 3 and requiring minJdk=11 > > seems > > > > > > > >>>>> reasonable > > > > > > > >>>>>> to > > > > > > > >>>>>>>> me. > > > > > > > >>>>>>>> > > > > > > > >>>>>>>> Gotta start pushing the issue somehow. > > > > > > > >>>>>>>> > > > > > > > >>>>>>>> On 2/15/22 1:47 PM, Sean Busbey wrote: > > > > > > > >>>>>>>>> Hi folks! > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>>> It's been some time since we decided to stick to LTS > > JDK > > > > > > > >>> releases > > > > > > > >>>>> as > > > > > > > >>>>>> a > > > > > > > >>>>>>>> way > > > > > > > >>>>>>>>> of getting a handle on the JDK treadmill. > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>>> What do folks think about deprecating JDK8? The > > openjdk8u > > > > > > > >>> project > > > > > > > >>>>> is > > > > > > > >>>>>>>> still > > > > > > > >>>>>>>>> going and there are commercial support options at > least > > > > > > > >> through > > > > > > > >>>>> 2030. > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>>> Deprecating it in HBase 3 would mean we could remove > it > > > in > > > > > > > >> HBase > > > > > > > >>>>> 4, > > > > > > > >>>>>> not > > > > > > > >>>>>>>>> that we would _have_ to remove it. The way I think > > about > > > > > > > >> likely > > > > > > > >>>>>> timing > > > > > > > >>>>>>> of > > > > > > > >>>>>>>>> these events goes like this: > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>>> * HBase 2 started alphas in June 2017, betas in > January > > > > 2018, > > > > > > > >>> and > > > > > > > >>>>>> came > > > > > > > >>>>>>>> out > > > > > > > >>>>>>>>> in April 2018 > > > > > > > >>>>>>>>> * HBase 3 started alphas in July 2021, and as of Feb > > 2022 > > > > we > > > > > > > >>>>> haven't > > > > > > > >>>>>>>>> discussed how close we are to our stated beta goals > > > > (upgrades > > > > > > > >>> from > > > > > > > >>>>>>> active > > > > > > > >>>>>>>>> 2.x releases and removal of not-ready features). > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>>> Given the above, in the absence of us specifically > > > pushing > > > > to > > > > > > > >>> roll > > > > > > > >>>>>>>> through > > > > > > > >>>>>>>>> major version numbers for some reason, I think a > > > reasonably > > > > > > > >>>>>>> conservative > > > > > > > >>>>>>>>> estimate is for HBase 3 to arrive in late 2022 or > early > > > > 2023 > > > > > > > >> and > > > > > > > >>>>> then > > > > > > > >>>>>>>> HBase > > > > > > > >>>>>>>>> 4 to start alphas in ~2025. An HBase 5 prior to 2030 > > > seems > > > > > > > >>>>> unlikely. > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>>> That all said, our current reference guide section on > > > java > > > > > > > >>>>> versions > > > > > > > >>>>>>> does > > > > > > > >>>>>>>>> not sound very confident about JDK11 support. > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>>>> A Note on JDK11 * > > > > > > > >>>>>>>>>> Preliminary support for JDK11 is introduced with > HBase > > > > > 2.3.0. > > > > > > > >>>>> This > > > > > > > >>>>>>>>> support is limited to > > > > > > > >>>>>>>>>> compilation and running the full test suite. There > are > > > > open > > > > > > > >>>>>> questions > > > > > > > >>>>>>>>> regarding the runtime > > > > > > > >>>>>>>>>> compatibility of JDK11 with Apache ZooKeeper and > > Apache > > > > > > > >> Hadoop > > > > > > > >>>>>>>>> (HADOOP-15338). > > > > > > > >>>>>>>>>> Significantly, neither project has yet released a > > > version > > > > > > > >> with > > > > > > > >>>>>>> explicit > > > > > > > >>>>>>>>> runtime support for > > > > > > > >>>>>>>>>> JDK11. The remaining known issues in HBase are > > > catalogued > > > > in > > > > > > > >>>>>>>> HBASE-22972. > > > > > > > >>>>>>>>>> > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>>> Since that blurb was written, Hadoop has added JDK11 > > > > support > > > > > > > >> [1] > > > > > > > >>>>> as > > > > > > > >>>>>> has > > > > > > > >>>>>>>>> ZooKeeper[2]. As a part of buttoning up our JDK11 > > support > > > > we > > > > > > > >>> could > > > > > > > >>>>>>> update > > > > > > > >>>>>>>>> our minimum supported versions of these projects to > > match > > > > > that > > > > > > > >>>>>> support. > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>>> What do folks think? > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>>> [1]: > https://hadoop.apache.org/docs/r3.3.0/index.html > > > > > > > >>>>>>>>> [2]: > > > > > > > >>>>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>> > > > > > > > >>> > > > > > > > > > > > > > > > > https://zookeeper.apache.org/doc/r3.6.0/zookeeperAdmin.html#sc_systemReq > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>> > > > > > > > >>>>> > > > > > > > >>>> > > > > > > > >>> > > > > > > > >> > > > > > > > >> > > > > > > > >> -- > > > > > > > >> Best regards, > > > > > > > >> Andrew > > > > > > > >> > > > > > > > >> Unrest, ignorance distilled, nihilistic imbeciles - > > > > > > > >> It's what we’ve earned > > > > > > > >> Welcome, apocalypse, what’s taken you so long? > > > > > > > >> Bring us the fitting end that we’ve been counting on > > > > > > > >> - A23, Welcome, Apocalypse > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Best regards, > > > > > Andrew > > > > > > > > > > Unrest, ignorance distilled, nihilistic imbeciles - > > > > > It's what we’ve earned > > > > > Welcome, apocalypse, what’s taken you so long? > > > > > Bring us the fitting end that we’ve been counting on > > > > > - A23, Welcome, Apocalypse > > > > > > > > > > > > > > > > > > -- > > > Best regards, > > > Andrew > > > > > > Unrest, ignorance distilled, nihilistic imbeciles - > > > It's what we’ve earned > > > Welcome, apocalypse, what’s taken you so long? > > > Bring us the fitting end that we’ve been counting on > > > - A23, Welcome, Apocalypse > > > > > > > > -- > Best regards, > Andrew > > Unrest, ignorance distilled, nihilistic imbeciles - > It's what we’ve earned > Welcome, apocalypse, what’s taken you so long? > Bring us the fitting end that we’ve been counting on > - A23, Welcome, Apocalypse >
