First of all, thank you very much for diligently pursuing hbase 3 support, this would greatly help maintain compatibility with both hbase 2 and 3 for all of our major changes going forward. Now that the beta version is released, we can expect hbase 3.0.0 very soon.
> While the HBase docs does not talk about this, I will also check if it is > possible to use protoc 3.x > in the HBase 3 build. Shall we start a DISCUSS thread on dev@hbase? In the same thread, we can also request for any better alternative to using maven-replacer-plugin, this is going to be a challenge for many downstream projects anyway? Moving to using protobuf 3 for hbase 3 profile is going to be challenging, while we need to maintain compatibility with hbase 2 as well. > some of the classes we use (f.e. RPCController) are simply not > there in Hbase 2.x (in the shaded packages) You mean RpcController is not present in hbase-thirdparty shaded module? If it's becoming complicated, it's okay to not explore this route. > We have already made a pass to replace HBase 1.x APIs, but there is still > work to do here. I can correlate, I expected this :) Hopefully this cleanup would only help us make better progress. > I plan to put up a PR to replace these APIs as soon as 5.2 is released. So this is going to be a prerequisite to supporting hbase 3 profile, correct? > As soon as HBase 3 support lands, we should make sure to at least compile > Phoenix with HBase 3 for every PR, lest old API calls creep back. +1, this means that post commit builds on the master branch would run all tests for profiles: 2.4.0, 2.4, 2.5.0, 2.5 and 3.0.0 whereas PR builds would run all tests for profiles: 2.4 and 3.0.0, right? As a side note, shall we also move to using 2.5 as default profile? On Thu, Feb 1, 2024 at 12:11 AM Istvan Toth <st...@apache.org> wrote: > HI! > > I have spent a few weeks getting Phoenix working with HBase 3, and while > I'm not done, I am at the point where everything compiles and most unit > tests run. > > My VERY WIP patch is at https://github.com/apache/phoenix/pull/1815, in > case you want to check. > > I want to get the discussion started, as keeping the code in one branch > requires some rather ugly hacks. > > PROTOBUF CHANGES > > The biggest difference is that HBase 3 removes protobuf 2.5, and the > unshaded hbase-protocol classes, and moves everything to protobuf 3. While > in the grand scheme of things this is great, it is a serious challenge to > us. > > The solution suggested in the HBase docs is basically just performing text > replacement using the maven-replacer-plugin, and I could not find a way to > avoid this. > > This means that when building for HBase 3, we copy everything to a tmp > directory, do a string replacement on the protobuf and hbase-protocol > packages in the source, then add these to the build with the build-helper > plugin. > > Surprisingly, my Eclipse picks up these changes, and it IS possible to > debug the generated code with some minor inconveniences. > > I have also considered simply switching the shaded libraries for HBase 2 as > well, but some of the classes we use (f.e. RPCController) are simply not > there in Hbase 2.x (in the shaded packages), so I did not pursue this > direction. If you think we should explore this direction, then I can make > more effort here. > > While the HBase docs does not talk about this, I will also check if it is > possible to use protoc 3.x > in the HBase 3 build. > > OTHER API CHANGES > > There are some minor API changes that can be handled just fine with the > usual compatibility modules, but there are also quite a few removed APIs. > We have already made a pass to replace HBase 1.x APIs, but there is still > work to do here. > > The good news is that the replacements for the removed APIs also work with > HBase 2.x, so they can be replaced anytime. > > I plan to put up a PR to replace these APIs as soon as 5.2 is released. > This way the actual > HBase 3 patch will be much smaller and easier to rebase as needed. > > CI > > As soon as HBase 3 support lands, we should make sure to at least compile > Phoenix with HBase 3 for every PR, lest old API calls creep back. > > Looking forward to your suggestions, questions or comments > > Istvan >
