This seems to have been caused by forward synchronous RPC changes between Hbase 2 and 3 (where HBase 2 simulates non-inclusive start keys by adding a \00 byte and setting them as inclusive, while HBase 3 handles them natively. This is something that we can handle on the Phoenix side.
On the other hand, I've found a major and a minor breaking change in HBase 3: The big problem is HBASE-29843 <https://issues.apache.org/jira/browse/HBASE-29843> HBaseRPCController.setPriority(final TableName tn) is never called on branch-3+ Phoenix relies on setting it its own HBaseRPCController to set custom priorities, but HBase 3 has mostly abandoned using HBaseRPCController, leaving no working mechanism for Phoenix to use. HBase needs to restore using it to maintain compatibility. (or provide some alternative method) The smaller problem is HBASE-29834 <https://issues.apache.org/jira/browse/HBASE-29834> MILLIS_BETWEEN_NEXTS metric is not updated on branch-3+ This does not break Phoenix, it just happens to be detected by a Phoenix test. Istvan On Mon, Nov 3, 2025 at 10:22 AM Istvan Toth <[email protected]> wrote: > Sorry, I missed your reply eralier, Duo. > > I still haven't figured out the issue (though at least now I am now able > to run those tests from a debugger) > No ticket yet, as I'm still not sure which project to use. > > To repro this, we need to > * install a current HBase 3 snapshot locally from branch-3 head, > * check out my WIP Phoenix HBase 3 branch: > https://github.com/stoty/phoenix/tree/PHOENIX-7164-2025 > * then run a faling test: > > *mvn clean verify -am -pl phoenix-core > -Dit.test=BaseAggregateWithRegionMoves2IT#testGroupByArray > -Dhbase.profile=3.0 -Dtest=foo* > There are other similar tests failures, missing expected ResultSet rows > after aggressively moving around the table regions during the query. > > Istvan > > On Thu, Oct 16, 2025 at 3:49 PM 张铎(Duo Zhang) <[email protected]> > wrote: > >> Is there an issue in HBase or Phoenix for tracking this problem? >> >> Thanks. >> >> Istvan Toth <[email protected]> 于2025年10月16日周四 11:58写道： >> > >> > Quick heads up: >> > I'm investigating a regression in the Phoenix test suite where scans are >> > not reset when the processed region is moved, leading to missing data >> from >> > the scan. >> > >> > I am not positive yet that the issue is on the HBase side, will report >> here >> > (and open a ticket if needed) when I know more. >> > >> > Istvan >> > >> > On Sat, Sep 13, 2025 at 2:18 PM 张铎(Duo Zhang) <[email protected]> >> wrote: >> > >> > > Only 4 issues left on 3.0.0-beta-2 now, will wait for one more week to >> > > see if we can get these issues done, otherwise will move them out of >> > > 3.0.0. >> > > >> > > And the HBCK2 issue has been fixed. >> > > >> > > Plan to cut a new hbase-operator-tools release before cutting >> 3.0.0-beta-2. >> > > >> > > Thanks. >> > > >> > > 张铎(Duo Zhang) <[email protected]> 于2025年8月31日周日 22:57写道： >> > > > >> > > > I've finished one round of reviewing pending issues for >> 3.0.0-beta-2. >> > > > >> > > > There are still some necessary ones we need to get in before the >> final >> > > > 3.0.0 release, like make HBCK2 work against hbase3. >> > > > >> > > > Will keep working on them. >> > > > >> > > > Thanks. >> > > > >> > > > Nihal Jain <[email protected]> 于2025年6月19日周四 16:06写道： >> > > > > >> > > > > Yes, from hbase-thirdparty side, release is done and also >> consumed in >> > > hbase >> > > > > branch-2.5 and branch-2.6 and we are good here. >> > > > > >> > > > > On Wed, 18 Jun 2025 at 8:48 PM, 张铎(Duo Zhang) < >> [email protected]> >> > > wrote: >> > > > > >> > > > > > Oh, I think we should cut 2.6.3 and 2.5.12 too? Since we have >> already >> > > > > > made the new thirdparty release without jetty 12. >> > > > > > >> > > > > > Thanks. >> > > > > > >> > > > > > Nihal Jain <[email protected]> 于2025年6月18日周三 01:51写道： >> > > > > > > >> > > > > > > Ah yes, let me restart work for hbase thirdparty release for >> jetty >> > > 12 >> > > > > > soon. >> > > > > > > >> > > > > > > On Tue, 17 Jun 2025 at 10:37 PM, Istvan Toth >> > > <[email protected] >> > > > > > > >> > > > > > > wrote: >> > > > > > > >> > > > > > > > I think we should make sure we're using Jetty 12 in the next >> > > beta. ( >> > > > > > so we >> > > > > > > > need a thirdparty release for Jetty 12) >> > > > > > > > We could also review other dependencies, and bump the ones >> (if >> > > any) >> > > > > > that >> > > > > > > > were kept on old versions because of the Java 8 requirement. >> > > > > > > > >> > > > > > > > Istvan >> > > > > > > > >> > > > > > > > On Tue, Jun 17, 2025 at 11:08 AM 张铎(Duo Zhang) < >> > > [email protected]> >> > > > > > > > wrote: >> > > > > > > > >> > > > > > > > > The good news is that I've finally got a successful 10B >> ITBLL >> > > run. >> > > > > > > > > >> > > > > > > > > You can check HBASE-27919 for more details. >> > > > > > > > > >> > > > > > > > > Next let me cleanup the jira issues and nightly runs. >> > > > > > > > > >> > > > > > > > > This time I think we can see 3.0.0 out soon. >> > > > > > > > > >> > > > > > > > > Thanks. >> > > > > > > > > >> > > > > > > > > Nihal Jain <[email protected]> 于2025年3月12日周三 03:37写道： >> > > > > > > > > > >> > > > > > > > > > Hi Duo, >> > > > > > > > > > >> > > > > > > > > > Have you tried hbase-3.x on a kerberized env? I >> encountered >> > > > > > HBASE-29144 >> > > > > > > > > with kerberos. Posting here to just bring to attention. >> > > > > > > > > > >> > > > > > > > > > Regards >> > > > > > > > > > Nihal >> > > > > > > > > > >> > > > > > > > > > On 2025/03/08 01:49:12 "张铎(Duo Zhang)" wrote: >> > > > > > > > > > > A good news, cluster replication worked fine while >> rolling >> > > > > > upgrading >> > > > > > > > > in my test. >> > > > > > > > > > > >> > > > > > > > > > > Let me focus on the region replication stuff. After >> fixing >> > > this >> > > > > > > > issue, >> > > > > > > > > > > the only blocker is a stable 10B ITBLL run. >> > > > > > > > > > > >> > > > > > > > > > > Hope we could get 3.0.0 out soon. >> > > > > > > > > > > >> > > > > > > > > > > Thanks. >> > > > > > > > > > > >> > > > > > > > > > > 张铎(Duo Zhang) <[email protected]> 于2025年3月6日周四 >> > > 22:55写道： >> > > > > > > > > > > > >> > > > > > > > > > > > I've finished one round of rolling upgrading test. >> It >> > > worked >> > > > > > fine. >> > > > > > > > > > > > Please see HBASE-29161. >> > > > > > > > > > > > >> > > > > > > > > > > > But soon I found a problem when memstore >> replication is >> > > > > > enabled, >> > > > > > > > and >> > > > > > > > > > > > filed HBASE-29169. >> > > > > > > > > > > > >> > > > > > > > > > > > I will test upgrading with cluster replication too, >> to >> > > see if >> > > > > > there >> > > > > > > > > > > > are any other issues. >> > > > > > > > > > > > >> > > > > > > > > > > > Thanks. >> > > > > > > > > > > > >> > > > > > > > > > > > Istvan Toth <[email protected]> 于2025年1月20日周一 >> 14:24写道： >> > > > > > > > > > > > > >> > > > > > > > > > > > > For HBASE-28431, only the final step of adding the >> > > > > > Hadoop-less >> > > > > > > > > assembly is >> > > > > > > > > > > > > missing. >> > > > > > > > > > > > > >> > > > > > > > > > > > > Please review >> > > https://github.com/apache/hbase/pull/5766 if >> > > > > > you >> > > > > > > > > have the >> > > > > > > > > > > > > time. >> > > > > > > > > > > > > >> > > > > > > > > > > > > Istvan >> > > > > > > > > > > > > >> > > > > > > > > > > > > On Tue, Dec 31, 2024 at 2:43 PM 张铎(Duo Zhang) < >> > > > > > > > > [email protected]> wrote: >> > > > > > > > > > > > > >> > > > > > > > > > > > > > I've finished the first round of issue >> alignment, it >> > > took >> > > > > > > > > several months... >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > And I've created a release audit tool for 3.0.0 >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > https://github.com/apache9/hbase-release-audit >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > where we can have several ignore files, so we >> do not >> > > need >> > > > > > to >> > > > > > > > > manually >> > > > > > > > > > > > > > check the result every time. >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > So now, there are still several things to be >> done >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > 1. Make sure we have all the features and fixes >> we >> > > want to >> > > > > > land >> > > > > > > > > on >> > > > > > > > > > > > > > branch-3. For example, I think Istvan's assembly >> > > cleanup >> > > > > > should >> > > > > > > > > be >> > > > > > > > > > > > > > included. Please report other issues you want to >> > > land in >> > > > > > final >> > > > > > > > > 3.0.0. >> > > > > > > > > > > > > > 2. Several stable 10B ITBLL runs. I will try to >> see >> > > if I >> > > > > > can >> > > > > > > > get >> > > > > > > > > > > > > > enough resources but it will be great if others >> > > could help >> > > > > > > > here. >> > > > > > > > > > > > > > 3. Rolling upgrading test. I plan to do rolling >> > > upgrade >> > > > > > test >> > > > > > > > > with the >> > > > > > > > > > > > > > latest 2.6.x and 2.5.x releases. Since we have >> big >> > > changes >> > > > > > on >> > > > > > > > > > > > > > replication and region replica replication, I >> plan >> > > to have >> > > > > > > > tests >> > > > > > > > > with >> > > > > > > > > > > > > > these two features enabled too if possible. >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > Any other suggestions? >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > Thanks. >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > 张铎(Duo Zhang) <[email protected]> >> 于2024年9月19日周四 >> > > > > > 22:35写道： >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > And do we want to apply the same trick to >> > > MasterObserver >> > > > > > and >> > > > > > > > > > > > > > > RegionServerObserver? >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > For RegionObserver, we'd better do this >> because >> > > there are >> > > > > > > > > already >> > > > > > > > > > > > > > > methods in it that use the ObserverContext<? >> > > extends >> > > > > > > > > > > > > > > RegionCoprocessorEnvironment> declaration, >> which >> > > were >> > > > > > added >> > > > > > > > in >> > > > > > > > > > > > > > > HBASE-10378. >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > For other coprocessor observers, we do not >> have >> > > such >> > > > > > methods >> > > > > > > > > yet. >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Nick Dimiduk <[email protected]> >> 于2024年9月19日周四 >> > > > > > 22:03写道： >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > We do it here or we wait for 4.0. I'm >> inclined >> > > toward >> > > > > > doing >> > > > > > > > > it here >> > > > > > > > > > > > > > > > because (i think) we expect coprocessor >> > > > > > implementations to >> > > > > > > > > necessarily >> > > > > > > > > > > > > > > > recompile anyway. Though I admit I've not >> thought >> > > > > > through >> > > > > > > > > what >> > > > > > > > > > > > > > features are >> > > > > > > > > > > > > > > > enabled by this new generic signature. >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > On Thu, Sep 19, 2024 at 3:57 PM 张铎(Duo >> Zhang) < >> > > > > > > > > [email protected]> >> > > > > > > > > > > > > > wrote: >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > Filed HBASE-28862. >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > It will break old coprocessor >> implementations >> > > even if >> > > > > > > > they >> > > > > > > > > do not use >> > > > > > > > > > > > > > > > > any deprecated method. But since it does >> not >> > > change >> > > > > > any >> > > > > > > > > abilities, I >> > > > > > > > > > > > > > > > > think it is OK to include this in a major >> > > release as >> > > > > > we >> > > > > > > > > have already >> > > > > > > > > > > > > > > > > done bunch of breaking changes, like >> changing >> > > the >> > > > > > > > protobuf >> > > > > > > > > from >> > > > > > > > > > > > > > > > > unshaded one to shaded and relocated >> one... >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > Thanks. >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > 张铎(Duo Zhang) <[email protected]> >> > > 于2024年9月19日周四 >> > > > > > > > > 17:36写道： >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > Thanks for pointing this TODO out. >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > Let me file an issue abort this. For me >> I >> > > think we >> > > > > > > > > should do this >> > > > > > > > > > > > > > in >> > > > > > > > > > > > > > > > > 3.0.0. >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > Nick Dimiduk <[email protected]> >> > > 于2024年9月19日周四 >> > > > > > > > > 16:46写道： >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > Should we take the opportunity to >> upgrade >> > > the >> > > > > > generic >> > > > > > > > > > > > > > definitions in >> > > > > > > > > > > > > > > > > our >> > > > > > > > > > > > > > > > > > > coprocessor signatures as well? >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > >> > > >> https://github.com/apache/hbase/blob/edbb145a3f0c6847f444fefdadea5b82d02b0bdb/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java#L101 >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > On Tue, Sep 17, 2024 at 4:41 PM Duo >> Zhang < >> > > > > > > > > [email protected]> >> > > > > > > > > > > > > > wrote: >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > I've done several rounds of API >> cleanup >> > > in >> > > > > > > > > HBASE-24888 and now >> > > > > > > > > > > > > > for me >> > > > > > > > > > > > > > > > > > > > I think it is enough for making the >> new >> > > major >> > > > > > > > > release. >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > I've filed HBASE-28844 for aligning >> the >> > > commit >> > > > > > > > > history and jira >> > > > > > > > > > > > > > > > > > > > issues, it will be a huge project >> for a >> > > major >> > > > > > > > > release and may >> > > > > > > > > > > > > > take >> > > > > > > > > > > > > > > > > > > > several weeks. >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > After that, I think we are good to >> go. >> > > And I >> > > > > > will >> > > > > > > > > also try to >> > > > > > > > > > > > > > find >> > > > > > > > > > > > > > > > > > > > some resources to run several >> rounds of >> > > ITBLL >> > > > > > and >> > > > > > > > > also some >> > > > > > > > > > > > > > upgrading >> > > > > > > > > > > > > > > > > > > > tests. Finally, after everything is >> OK, >> > > I will >> > > > > > > > > release >> > > > > > > > > > > > > > 3.0.0-beta-2, >> > > > > > > > > > > > > > > > > > > > and then cut branch-3.0 from the >> > > 3.0.0-beta-2 >> > > > > > tag >> > > > > > > > > and make the >> > > > > > > > > > > > > > 3.0.0 >> > > > > > > > > > > > > > > > > > > > release soon. >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > Hope I can get all these things done >> > > within the >> > > > > > > > year >> > > > > > > > > 2024. >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > Thanks. >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > -- >> > > > > > > > *István Tóth* | Sr. Staff Software Engineer >> > > > > > > > *Email*: [email protected] >> > > > > > > > cloudera.com <https://www.cloudera.com> >> > > > > > > > [image: Cloudera] <https://www.cloudera.com/> >> > > > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> >> > > [image: >> > > > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> >> > > [image: >> > > > > > Cloudera >> > > > > > > > on LinkedIn] <https://www.linkedin.com/company/cloudera> >> > > > > > > > ------------------------------ >> > > > > > > > ------------------------------ >> > > > > > > > >> > > > > > >> > > >> > >> > >> > -- >> > *István Tóth* | Sr. Staff Software Engineer >> > *Email*: [email protected] >> > cloudera.com <https://www.cloudera.com> >> > [image: Cloudera] <https://www.cloudera.com/> >> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: >> > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: >> Cloudera >> > on LinkedIn] <https://www.linkedin.com/company/cloudera> >> > ------------------------------ >> > ------------------------------ >> > > > -- > *István Tóth* | Sr. Staff Software Engineer > *Email*: [email protected] > cloudera.com <https://www.cloudera.com> > [image: Cloudera] <https://www.cloudera.com/> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> > ------------------------------ > ------------------------------ > -- *István Tóth* | Sr. Staff Software Engineer *Email*: [email protected] cloudera.com <https://www.cloudera.com> [image: Cloudera] <https://www.cloudera.com/> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> ------------------------------ ------------------------------
