We often upgrade netty after there is a CVE announcement. This produces a pressure to upgrade in production. Use of an internal API is not a good practice because it can cause friction because of breaking changes when there is a need to upgrade all of a sudden. This is painful and hopefully we can exclude it as a possibility now when coming up with the plan. We can do the same thing they do in our own hbase-thirdparty without taking the dependency and achieve the same goal, right?
On Mon, Feb 21, 2022 at 7:35 PM 张铎(Duo Zhang) <palomino...@gmail.com> wrote: > For me, since we shade netty, there will be no conflict with other netty > dependencies, so we can make sure the netty version we used is exactly the > one we shaded in hbase-thirdparty. > > If later we want to upgrade netty version in hbase-thirdparty, and it > breaks the api, then we can release a new hbase-thirdparty, and upgrade the > main hbase repo to depend on the new hbase-thirdparty version, and also > modify the code in hbase accordingly. There will be no user visible > affects. > > Anyway, let me have a try first. Even with netty's PaltformDependent class, > since it still references sun.misc.Unsafe, maybe we will still get an > compile error... > > Thanks. > > Andrew Purtell <apurt...@apache.org> 于2022年2月22日周二 08:08写道： > > > Thanks for that. I understand better what you are proposing now. It is > > clear that you have thought it through. > > > > So then the only question I have is this: Do you think it is a concern > that > > Netty's PlatformDependent class is in their "io.netty.util.internal" > > package? The 'internal' designation suggests to me they won't apply the > > same care to not breaking users of it as they would for their public > > (non-"internal") API. I noticed this earlier but had other questions at > > that time. > > > > If it is a concern, we have the option of doing the same thing that Netty > > has done with their PlatformDependent class as a new utility class of our > > own in hbase-thirdparty. This may impose special requirements on building > > hbase-thirdparty, or, at least, the module containing this wrapper, but > it > > is an option that would help us avoid external breaking changes. For your > > consideration. > > > > On Mon, Feb 21, 2022 at 3:48 PM 张铎(Duo Zhang) <palomino...@gmail.com> > > wrote: > > > > > What I proposed here is for solving the problem… > > > > > > The problem is that when using —release 8, javac will not export the > > > jdk.unsupported symbols, so using sun.misc.Unsafe will trigger a > compile > > > error. But at runtime if you export jdk.unsupported you can still use > > > sun.misc.Unsafe. The related jdk issue said —release is only for public > > API > > > so not exporting jdk.unsupported is the expected behavior. > > > > > > So what I proposed here is to remove direct dependency on > sun.misc.Unsafe > > > in HBase code, so at compile time there will be no problem. Betty has a > > > wrapper of Unsafe so I think we could have a try. At runtime if we have > > > jdk.unsupported exported we could still actually make use of > > > sun.misc.Unsafe. > > > > > > Feel free to correct me if I missed something or there are still > > uncleared > > > things. > > > > > > Thanks. > > > > > > > > > > > > Andrew Purtell <andrew.purt...@gmail.com>于2022年2月22日 周二00:56写道： > > > > > > > As Nick discovered ‘—release’ doesn’t work for version 8 anyway. > > > > > > > > > > > > > On Feb 20, 2022, at 3:47 PM, 张铎 <palomino...@gmail.com> wrote: > > > > > > > > > > I know, with —release 8 ByteBuffer will not be an problem but then > > > > > sun.misc.Unsafe can not be used any more right? So the problem here > > is > > > > how > > > > > to make use of Unsafe with —release 8. What I proposed here is that > > we > > > > just > > > > > do not use it, then we can make use of —release 8 to address the > > > > ByteBuffer > > > > > problem. > > > > > > > > > > Thanks. > > > > > > > > > > Andrew Purtell <apurt...@apache.org>于2022年2月21日 周一03:32写道： > > > > > > > > > >> The problem that leads us to consider '--release 8', or what it > > would > > > > >> promise if it worked, isn't Unsafe, it is ByteBuffer. > > > > >> > > > > > > > > > > https://www.morling.dev/blog/bytebuffer-and-the-dreaded-nosuchmethoderror/ > > > > >> . Although to your point Netty has ByteBuf... > > > > >> > > > > >> > > > > >>> On Sun, Feb 20, 2022 at 5:23 AM 张铎(Duo Zhang) < > > palomino...@gmail.com > > > > > > > > >>> wrote: > > > > >>> > > > > >>> Since we have shaded netty, I think we could make use of > > > > >>> netty's PlatformDependent[1] class to avoid referencing Unsafe > > > > directly, > > > > >>> then there will be no problem for us to use the '--release 8' > > option. > > > > >>> > > > > >>> If no big concerns, I will file an issue to remove all the > > > > >> sun.misc.Unsafe > > > > >>> references in our code base and make use of PlatformDependent > from > > > the > > > > >>> shaded netty. > > > > >>> > > > > >>> Thanks. > > > > >>> > > > > >>> 1. > > > > >>> > > > > >>> > > > > >> > > > > > > > > > > https://github.com/netty/netty/blob/4.1/common/src/main/java/io/netty/util/internal/PlatformDependent.java > > > > >>> > > > > >>> 张铎(Duo Zhang) <palomino...@gmail.com> 于2022年2月16日周三 14:12写道： > > > > >>> > > > > >>>> I think this could be done by some module tricks, where we > build a > > > > >>> special > > > > >>>> module with -source 8 and -target 8.We have done similar things > in > > > the > > > > >>> past > > > > >>>> to have hadoop1-compat and hadoop2-compact. > > > > >>>> > > > > >>>> Let me have a try. > > > > >>>> > > > > >>>> Sean Busbey <bus...@apache.org> 于2022年2月16日周三 13:31写道： > > > > >>>> > > > > >>>>> Unfortunately if we want to keep jdk8 working we can't rely on > > > > >> building > > > > >>>>> with the jdk release target option on newer jdks. > > > > >>>>> > > > > >>>>> We ran into this recently in HBASE-25465 where a JDK bug came > up. > > > > >>>>> > > > > >>>>> On Tue, Feb 15, 2022 at 8:12 PM 张铎(Duo Zhang) < > > > palomino...@gmail.com > > > > > > > > > >>>>> wrote: > > > > >>>>> > > > > >>>>>> +1 on moving the minimum required java version for compiling > > HBase > > > > >> to > > > > >>>>> Java > > > > >>>>>> 11. But we could still use --release=8 to still support jdk8 > at > > > > >>> runtime. > > > > >>>>>> > > > > >>>>>> Java 8 is still widely used, and I believe this will last for > > even > > > > >>> more > > > > >>>>>> years, as lots of big companies such as RedHat, Microsoft and > > > Amazon > > > > >>> are > > > > >>>>>> still shipping new jdk8 releases, I do not think it is time to > > > fully > > > > >>>>> drop > > > > >>>>>> the support of jdk8. > > > > >>>>>> > > > > >>>>>> Thanks. > > > > >>>>>> > > > > >>>>>> Sean Busbey <bus...@apache.org> 于2022年2月16日周三 09:57写道： > > > > >>>>>> > > > > >>>>>>> If we set the minJDK to 11 I believe that will effectively > > remove > > > > >>> jdk8 > > > > >>>>>>> support, rather than "just" deprecate it. > > > > >>>>>>> > > > > >>>>>>> As we build out more robust testing I would be in favor of > > > running > > > > >>>>> less > > > > >>>>>> of > > > > >>>>>>> it on deprecated JDKs. > > > > >>>>>>> > > > > >>>>>>> > > > > >>>>>>> On Tue, Feb 15, 2022, 16:10 Josh Elser <els...@apache.org> > > > wrote: > > > > >>>>>>> > > > > >>>>>>>> Deprecating jdk8 for HBase 3 and requiring minJdk=11 seems > > > > >>>>> reasonable > > > > >>>>>> to > > > > >>>>>>>> me. > > > > >>>>>>>> > > > > >>>>>>>> Gotta start pushing the issue somehow. > > > > >>>>>>>> > > > > >>>>>>>> On 2/15/22 1:47 PM, Sean Busbey wrote: > > > > >>>>>>>>> Hi folks! > > > > >>>>>>>>> > > > > >>>>>>>>> It's been some time since we decided to stick to LTS JDK > > > > >>> releases > > > > >>>>> as > > > > >>>>>> a > > > > >>>>>>>> way > > > > >>>>>>>>> of getting a handle on the JDK treadmill. > > > > >>>>>>>>> > > > > >>>>>>>>> What do folks think about deprecating JDK8? The openjdk8u > > > > >>> project > > > > >>>>> is > > > > >>>>>>>> still > > > > >>>>>>>>> going and there are commercial support options at least > > > > >> through > > > > >>>>> 2030. > > > > >>>>>>>>> > > > > >>>>>>>>> Deprecating it in HBase 3 would mean we could remove it in > > > > >> HBase > > > > >>>>> 4, > > > > >>>>>> not > > > > >>>>>>>>> that we would _have_ to remove it. The way I think about > > > > >> likely > > > > >>>>>> timing > > > > >>>>>>> of > > > > >>>>>>>>> these events goes like this: > > > > >>>>>>>>> > > > > >>>>>>>>> * HBase 2 started alphas in June 2017, betas in January > 2018, > > > > >>> and > > > > >>>>>> came > > > > >>>>>>>> out > > > > >>>>>>>>> in April 2018 > > > > >>>>>>>>> * HBase 3 started alphas in July 2021, and as of Feb 2022 > we > > > > >>>>> haven't > > > > >>>>>>>>> discussed how close we are to our stated beta goals > (upgrades > > > > >>> from > > > > >>>>>>> active > > > > >>>>>>>>> 2.x releases and removal of not-ready features). > > > > >>>>>>>>> > > > > >>>>>>>>> Given the above, in the absence of us specifically pushing > to > > > > >>> roll > > > > >>>>>>>> through > > > > >>>>>>>>> major version numbers for some reason, I think a reasonably > > > > >>>>>>> conservative > > > > >>>>>>>>> estimate is for HBase 3 to arrive in late 2022 or early > 2023 > > > > >> and > > > > >>>>> then > > > > >>>>>>>> HBase > > > > >>>>>>>>> 4 to start alphas in ~2025. An HBase 5 prior to 2030 seems > > > > >>>>> unlikely. > > > > >>>>>>>>> > > > > >>>>>>>>> That all said, our current reference guide section on java > > > > >>>>> versions > > > > >>>>>>> does > > > > >>>>>>>>> not sound very confident about JDK11 support. > > > > >>>>>>>>> > > > > >>>>>>>>>> A Note on JDK11 * > > > > >>>>>>>>>> Preliminary support for JDK11 is introduced with HBase > > 2.3.0. > > > > >>>>> This > > > > >>>>>>>>> support is limited to > > > > >>>>>>>>>> compilation and running the full test suite. There are > open > > > > >>>>>> questions > > > > >>>>>>>>> regarding the runtime > > > > >>>>>>>>>> compatibility of JDK11 with Apache ZooKeeper and Apache > > > > >> Hadoop > > > > >>>>>>>>> (HADOOP-15338). > > > > >>>>>>>>>> Significantly, neither project has yet released a version > > > > >> with > > > > >>>>>>> explicit > > > > >>>>>>>>> runtime support for > > > > >>>>>>>>>> JDK11. The remaining known issues in HBase are catalogued > in > > > > >>>>>>>> HBASE-22972. > > > > >>>>>>>>>> > > > > >>>>>>>>> > > > > >>>>>>>>> Since that blurb was written, Hadoop has added JDK11 > support > > > > >> [1] > > > > >>>>> as > > > > >>>>>> has > > > > >>>>>>>>> ZooKeeper[2]. As a part of buttoning up our JDK11 support > we > > > > >>> could > > > > >>>>>>> update > > > > >>>>>>>>> our minimum supported versions of these projects to match > > that > > > > >>>>>> support. > > > > >>>>>>>>> > > > > >>>>>>>>> What do folks think? > > > > >>>>>>>>> > > > > >>>>>>>>> [1]: https://hadoop.apache.org/docs/r3.3.0/index.html > > > > >>>>>>>>> [2]: > > > > >>>>>>>>> > > > > >>>>>>> > > > > >>>>> > > > > >>> > > > > > > https://zookeeper.apache.org/doc/r3.6.0/zookeeperAdmin.html#sc_systemReq > > > > >>>>>>>>> > > > > >>>>>>>> > > > > >>>>>>> > > > > >>>>>> > > > > >>>>> > > > > >>>> > > > > >>> > > > > >> > > > > >> > > > > >> -- > > > > >> Best regards, > > > > >> Andrew > > > > >> > > > > >> Unrest, ignorance distilled, nihilistic imbeciles - > > > > >> It's what we’ve earned > > > > >> Welcome, apocalypse, what’s taken you so long? > > > > >> Bring us the fitting end that we’ve been counting on > > > > >> - A23, Welcome, Apocalypse > > > > >> > > > > > > > > > > > > > -- > > Best regards, > > Andrew > > > > Unrest, ignorance distilled, nihilistic imbeciles - > > It's what we’ve earned > > Welcome, apocalypse, what’s taken you so long? > > Bring us the fitting end that we’ve been counting on > > - A23, Welcome, Apocalypse > > > -- Best regards, Andrew Unrest, ignorance distilled, nihilistic imbeciles - It's what we’ve earned Welcome, apocalypse, what’s taken you so long? Bring us the fitting end that we’ve been counting on - A23, Welcome, Apocalypse
