Thanks for your reply, Viraj, On Fri, Feb 2, 2024 at 5:45 AM Viraj Jasani <vjas...@apache.org> wrote:
> First of all, thank you very much for diligently pursuing hbase 3 support, > this would greatly help maintain compatibility with both hbase 2 and 3 for > all of our major changes going forward. Now that the beta version is > released, we can expect hbase 3.0.0 very soon. > > > While the HBase docs does not talk about this, I will also check if it is > > possible to use protoc 3.x > > in the HBase 3 build. > > Shall we start a DISCUSS thread on dev@hbase? In the same thread, we can > also request for any better alternative to using maven-replacer-plugin, > this is going to be a challenge for many downstream projects anyway? > Moving to using protobuf 3 for hbase 3 profile is going to be challenging, > while we need to maintain compatibility with hbase 2 as well. > I plan to do some experiments first, I would expect it to work, as my POC already uses 3.x runtime libraries ( with the 2.5 generated code) and HBase also uses protoc 3 internally. Still, there is a possibility that the current phoenix .protoc files are not handled by the 3.x protoc. I know that some features were removed in protoc 3, but we may not be using them. The replacer plugin works fine, it's just a huge hack. The only other option I can see is adding a layer of indirection for everything protocol and protobuf related into compatibility modules, but I feel that would be even worse. I am afraid that replacer-plugin really IS the best option. It doesn't hurt to ask, though. It only affects downstream projects that define HBase endpoints and/or directly use the protobuf API. > > some of the classes we use (f.e. RPCController) are simply not > > there in Hbase 2.x (in the shaded packages) > > You mean RpcController is not present in hbase-thirdparty shaded module? If > it's becoming complicated, it's okay to not explore this route. > > Hmm. RPCControllers ARE present in 3.0. Maybe I have just missed something. I will explore this some more, and report back. > We have already made a pass to replace HBase 1.x APIs, but there is still > > work to do here. > > I can correlate, I expected this :) > Hopefully this cleanup would only help us make better progress. > > > I plan to put up a PR to replace these APIs as soon as 5.2 is released. > > So this is going to be a prerequisite to supporting hbase 3 profile, > correct? > Absolutely. Only we don't have to wait for the release to do this. > > > As soon as HBase 3 support lands, we should make sure to at least compile > > Phoenix with HBase 3 for every PR, lest old API calls creep back. > > +1, this means that post commit builds on the master branch would run all > tests for profiles: 2.4.0, 2.4, 2.5.0, 2.5 and 3.0.0 whereas PR builds > would run all tests for profiles: 2.4 and 3.0.0, right? > > That's the plan. > As a side note, shall we also move to using 2.5 as default profile? > Looking at the HBase download page, 2.5 is marked as the stable release now, so yes. We could do that now, in fact. > > On Thu, Feb 1, 2024 at 12:11 AM Istvan Toth <st...@apache.org> wrote: > > > HI! > > > > I have spent a few weeks getting Phoenix working with HBase 3, and while > > I'm not done, I am at the point where everything compiles and most unit > > tests run. > > > > My VERY WIP patch is at https://github.com/apache/phoenix/pull/1815, in > > case you want to check. > > > > I want to get the discussion started, as keeping the code in one branch > > requires some rather ugly hacks. > > > > PROTOBUF CHANGES > > > > The biggest difference is that HBase 3 removes protobuf 2.5, and the > > unshaded hbase-protocol classes, and moves everything to protobuf 3. > While > > in the grand scheme of things this is great, it is a serious challenge to > > us. > > > > The solution suggested in the HBase docs is basically just performing > text > > replacement using the maven-replacer-plugin, and I could not find a way > to > > avoid this. > > > > This means that when building for HBase 3, we copy everything to a tmp > > directory, do a string replacement on the protobuf and hbase-protocol > > packages in the source, then add these to the build with the build-helper > > plugin. > > > > Surprisingly, my Eclipse picks up these changes, and it IS possible to > > debug the generated code with some minor inconveniences. > > > > I have also considered simply switching the shaded libraries for HBase 2 > as > > well, but some of the classes we use (f.e. RPCController) are simply not > > there in Hbase 2.x (in the shaded packages), so I did not pursue this > > direction. If you think we should explore this direction, then I can make > > more effort here. > > > > While the HBase docs does not talk about this, I will also check if it is > > possible to use protoc 3.x > > in the HBase 3 build. > > > > OTHER API CHANGES > > > > There are some minor API changes that can be handled just fine with the > > usual compatibility modules, but there are also quite a few removed APIs. > > We have already made a pass to replace HBase 1.x APIs, but there is still > > work to do here. > > > > The good news is that the replacements for the removed APIs also work > with > > HBase 2.x, so they can be replaced anytime. > > > > I plan to put up a PR to replace these APIs as soon as 5.2 is released. > > This way the actual > > HBase 3 patch will be much smaller and easier to rebase as needed. > > > > CI > > > > As soon as HBase 3 support lands, we should make sure to at least compile > > Phoenix with HBase 3 for every PR, lest old API calls creep back. > > > > Looking forward to your suggestions, questions or comments > > > > Istvan > > > -- *István Tóth* | Sr. Staff Software Engineer *Email*: st...@cloudera.com cloudera.com <https://www.cloudera.com> [image: Cloudera] <https://www.cloudera.com/> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> ------------------------------ ------------------------------
