Thanks Istvan for the detailed analysis! For HBASE-29834, I've already put up a PR to fix the problem. See
https://github.com/apache/hbase/pull/7660 And on HBASE-29843, I do not fully understand the problem and need more context about the usage on Phoenix side. I think we should be able to fix it. Thanks. Istvan Toth <[email protected]> 于2026年1月22日周四 14:33写道： > > This seems to have been caused by forward synchronous RPC changes between > Hbase 2 and 3 (where HBase 2 simulates non-inclusive start keys by > adding a \00 byte and setting them as inclusive, while HBase 3 handles them > natively. > This is something that we can handle on the Phoenix side. > > On the other hand, I've found a major and a minor breaking change in HBase > 3: > > The big problem is HBASE-29843 > <https://issues.apache.org/jira/browse/HBASE-29843> > HBaseRPCController.setPriority(final TableName tn) is never called on > branch-3+ > Phoenix relies on setting it its own HBaseRPCController to set custom > priorities, but HBase 3 has mostly abandoned using HBaseRPCController, > leaving no > working mechanism for Phoenix to use. > HBase needs to restore using it to maintain compatibility. (or provide some > alternative method) > > The smaller problem is HBASE-29834 > <https://issues.apache.org/jira/browse/HBASE-29834> MILLIS_BETWEEN_NEXTS > metric is not updated on branch-3+ > This does not break Phoenix, it just happens to be detected by a Phoenix > test. > > Istvan > > > On Mon, Nov 3, 2025 at 10:22 AM Istvan Toth <[email protected]> wrote: > > > Sorry, I missed your reply eralier, Duo. > > > > I still haven't figured out the issue (though at least now I am now able > > to run those tests from a debugger) > > No ticket yet, as I'm still not sure which project to use. > > > > To repro this, we need to > > * install a current HBase 3 snapshot locally from branch-3 head, > > * check out my WIP Phoenix HBase 3 branch: > > https://github.com/stoty/phoenix/tree/PHOENIX-7164-2025 > > * then run a faling test: > > > > *mvn clean verify -am -pl phoenix-core > > -Dit.test=BaseAggregateWithRegionMoves2IT#testGroupByArray > > -Dhbase.profile=3.0 -Dtest=foo* > > There are other similar tests failures, missing expected ResultSet rows > > after aggressively moving around the table regions during the query. > > > > Istvan > > > > On Thu, Oct 16, 2025 at 3:49 PM 张铎(Duo Zhang) <[email protected]> > > wrote: > > > >> Is there an issue in HBase or Phoenix for tracking this problem? > >> > >> Thanks. > >> > >> Istvan Toth <[email protected]> 于2025年10月16日周四 11:58写道： > >> > > >> > Quick heads up: > >> > I'm investigating a regression in the Phoenix test suite where scans are > >> > not reset when the processed region is moved, leading to missing data > >> from > >> > the scan. > >> > > >> > I am not positive yet that the issue is on the HBase side, will report > >> here > >> > (and open a ticket if needed) when I know more. > >> > > >> > Istvan > >> > > >> > On Sat, Sep 13, 2025 at 2:18 PM 张铎(Duo Zhang) <[email protected]> > >> wrote: > >> > > >> > > Only 4 issues left on 3.0.0-beta-2 now, will wait for one more week to > >> > > see if we can get these issues done, otherwise will move them out of > >> > > 3.0.0. > >> > > > >> > > And the HBCK2 issue has been fixed. > >> > > > >> > > Plan to cut a new hbase-operator-tools release before cutting > >> 3.0.0-beta-2. > >> > > > >> > > Thanks. > >> > > > >> > > 张铎(Duo Zhang) <[email protected]> 于2025年8月31日周日 22:57写道： > >> > > > > >> > > > I've finished one round of reviewing pending issues for > >> 3.0.0-beta-2. > >> > > > > >> > > > There are still some necessary ones we need to get in before the > >> final > >> > > > 3.0.0 release, like make HBCK2 work against hbase3. > >> > > > > >> > > > Will keep working on them. > >> > > > > >> > > > Thanks. > >> > > > > >> > > > Nihal Jain <[email protected]> 于2025年6月19日周四 16:06写道： > >> > > > > > >> > > > > Yes, from hbase-thirdparty side, release is done and also > >> consumed in > >> > > hbase > >> > > > > branch-2.5 and branch-2.6 and we are good here. > >> > > > > > >> > > > > On Wed, 18 Jun 2025 at 8:48 PM, 张铎(Duo Zhang) < > >> [email protected]> > >> > > wrote: > >> > > > > > >> > > > > > Oh, I think we should cut 2.6.3 and 2.5.12 too? Since we have > >> already > >> > > > > > made the new thirdparty release without jetty 12. > >> > > > > > > >> > > > > > Thanks. > >> > > > > > > >> > > > > > Nihal Jain <[email protected]> 于2025年6月18日周三 01:51写道： > >> > > > > > > > >> > > > > > > Ah yes, let me restart work for hbase thirdparty release for > >> jetty > >> > > 12 > >> > > > > > soon. > >> > > > > > > > >> > > > > > > On Tue, 17 Jun 2025 at 10:37 PM, Istvan Toth > >> > > <[email protected] > >> > > > > > > > >> > > > > > > wrote: > >> > > > > > > > >> > > > > > > > I think we should make sure we're using Jetty 12 in the next > >> > > beta. ( > >> > > > > > so we > >> > > > > > > > need a thirdparty release for Jetty 12) > >> > > > > > > > We could also review other dependencies, and bump the ones > >> (if > >> > > any) > >> > > > > > that > >> > > > > > > > were kept on old versions because of the Java 8 requirement. > >> > > > > > > > > >> > > > > > > > Istvan > >> > > > > > > > > >> > > > > > > > On Tue, Jun 17, 2025 at 11:08 AM 张铎(Duo Zhang) < > >> > > [email protected]> > >> > > > > > > > wrote: > >> > > > > > > > > >> > > > > > > > > The good news is that I've finally got a successful 10B > >> ITBLL > >> > > run. > >> > > > > > > > > > >> > > > > > > > > You can check HBASE-27919 for more details. > >> > > > > > > > > > >> > > > > > > > > Next let me cleanup the jira issues and nightly runs. > >> > > > > > > > > > >> > > > > > > > > This time I think we can see 3.0.0 out soon. > >> > > > > > > > > > >> > > > > > > > > Thanks. > >> > > > > > > > > > >> > > > > > > > > Nihal Jain <[email protected]> 于2025年3月12日周三 03:37写道： > >> > > > > > > > > > > >> > > > > > > > > > Hi Duo, > >> > > > > > > > > > > >> > > > > > > > > > Have you tried hbase-3.x on a kerberized env? I > >> encountered > >> > > > > > HBASE-29144 > >> > > > > > > > > with kerberos. Posting here to just bring to attention. > >> > > > > > > > > > > >> > > > > > > > > > Regards > >> > > > > > > > > > Nihal > >> > > > > > > > > > > >> > > > > > > > > > On 2025/03/08 01:49:12 "张铎(Duo Zhang)" wrote: > >> > > > > > > > > > > A good news, cluster replication worked fine while > >> rolling > >> > > > > > upgrading > >> > > > > > > > > in my test. > >> > > > > > > > > > > > >> > > > > > > > > > > Let me focus on the region replication stuff. After > >> fixing > >> > > this > >> > > > > > > > issue, > >> > > > > > > > > > > the only blocker is a stable 10B ITBLL run. > >> > > > > > > > > > > > >> > > > > > > > > > > Hope we could get 3.0.0 out soon. > >> > > > > > > > > > > > >> > > > > > > > > > > Thanks. > >> > > > > > > > > > > > >> > > > > > > > > > > 张铎(Duo Zhang) <[email protected]> 于2025年3月6日周四 > >> > > 22:55写道： > >> > > > > > > > > > > > > >> > > > > > > > > > > > I've finished one round of rolling upgrading test. > >> It > >> > > worked > >> > > > > > fine. > >> > > > > > > > > > > > Please see HBASE-29161. > >> > > > > > > > > > > > > >> > > > > > > > > > > > But soon I found a problem when memstore > >> replication is > >> > > > > > enabled, > >> > > > > > > > and > >> > > > > > > > > > > > filed HBASE-29169. > >> > > > > > > > > > > > > >> > > > > > > > > > > > I will test upgrading with cluster replication too, > >> to > >> > > see if > >> > > > > > there > >> > > > > > > > > > > > are any other issues. > >> > > > > > > > > > > > > >> > > > > > > > > > > > Thanks. > >> > > > > > > > > > > > > >> > > > > > > > > > > > Istvan Toth <[email protected]> 于2025年1月20日周一 > >> 14:24写道： > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > For HBASE-28431, only the final step of adding the > >> > > > > > Hadoop-less > >> > > > > > > > > assembly is > >> > > > > > > > > > > > > missing. > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > Please review > >> > > https://github.com/apache/hbase/pull/5766 if > >> > > > > > you > >> > > > > > > > > have the > >> > > > > > > > > > > > > time. > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > Istvan > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > On Tue, Dec 31, 2024 at 2:43 PM 张铎(Duo Zhang) < > >> > > > > > > > > [email protected]> wrote: > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > I've finished the first round of issue > >> alignment, it > >> > > took > >> > > > > > > > > several months... > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > And I've created a release audit tool for 3.0.0 > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > https://github.com/apache9/hbase-release-audit > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > where we can have several ignore files, so we > >> do not > >> > > need > >> > > > > > to > >> > > > > > > > > manually > >> > > > > > > > > > > > > > check the result every time. > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > So now, there are still several things to be > >> done > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > 1. Make sure we have all the features and fixes > >> we > >> > > want to > >> > > > > > land > >> > > > > > > > > on > >> > > > > > > > > > > > > > branch-3. For example, I think Istvan's assembly > >> > > cleanup > >> > > > > > should > >> > > > > > > > > be > >> > > > > > > > > > > > > > included. Please report other issues you want to > >> > > land in > >> > > > > > final > >> > > > > > > > > 3.0.0. > >> > > > > > > > > > > > > > 2. Several stable 10B ITBLL runs. I will try to > >> see > >> > > if I > >> > > > > > can > >> > > > > > > > get > >> > > > > > > > > > > > > > enough resources but it will be great if others > >> > > could help > >> > > > > > > > here. > >> > > > > > > > > > > > > > 3. Rolling upgrading test. I plan to do rolling > >> > > upgrade > >> > > > > > test > >> > > > > > > > > with the > >> > > > > > > > > > > > > > latest 2.6.x and 2.5.x releases. Since we have > >> big > >> > > changes > >> > > > > > on > >> > > > > > > > > > > > > > replication and region replica replication, I > >> plan > >> > > to have > >> > > > > > > > tests > >> > > > > > > > > with > >> > > > > > > > > > > > > > these two features enabled too if possible. > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > Any other suggestions? > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > Thanks. > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > 张铎(Duo Zhang) <[email protected]> > >> 于2024年9月19日周四 > >> > > > > > 22:35写道： > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > And do we want to apply the same trick to > >> > > MasterObserver > >> > > > > > and > >> > > > > > > > > > > > > > > RegionServerObserver? > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > For RegionObserver, we'd better do this > >> because > >> > > there are > >> > > > > > > > > already > >> > > > > > > > > > > > > > > methods in it that use the ObserverContext<? > >> > > extends > >> > > > > > > > > > > > > > > RegionCoprocessorEnvironment> declaration, > >> which > >> > > were > >> > > > > > added > >> > > > > > > > in > >> > > > > > > > > > > > > > > HBASE-10378. > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > For other coprocessor observers, we do not > >> have > >> > > such > >> > > > > > methods > >> > > > > > > > > yet. > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Nick Dimiduk <[email protected]> > >> 于2024年9月19日周四 > >> > > > > > 22:03写道： > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > We do it here or we wait for 4.0. I'm > >> inclined > >> > > toward > >> > > > > > doing > >> > > > > > > > > it here > >> > > > > > > > > > > > > > > > because (i think) we expect coprocessor > >> > > > > > implementations to > >> > > > > > > > > necessarily > >> > > > > > > > > > > > > > > > recompile anyway. Though I admit I've not > >> thought > >> > > > > > through > >> > > > > > > > > what > >> > > > > > > > > > > > > > features are > >> > > > > > > > > > > > > > > > enabled by this new generic signature. > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > On Thu, Sep 19, 2024 at 3:57 PM 张铎(Duo > >> Zhang) < > >> > > > > > > > > [email protected]> > >> > > > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > Filed HBASE-28862. > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > It will break old coprocessor > >> implementations > >> > > even if > >> > > > > > > > they > >> > > > > > > > > do not use > >> > > > > > > > > > > > > > > > > any deprecated method. But since it does > >> not > >> > > change > >> > > > > > any > >> > > > > > > > > abilities, I > >> > > > > > > > > > > > > > > > > think it is OK to include this in a major > >> > > release as > >> > > > > > we > >> > > > > > > > > have already > >> > > > > > > > > > > > > > > > > done bunch of breaking changes, like > >> changing > >> > > the > >> > > > > > > > protobuf > >> > > > > > > > > from > >> > > > > > > > > > > > > > > > > unshaded one to shaded and relocated > >> one... > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > Thanks. > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > 张铎(Duo Zhang) <[email protected]> > >> > > 于2024年9月19日周四 > >> > > > > > > > > 17:36写道： > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > Thanks for pointing this TODO out. > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > Let me file an issue abort this. For me > >> I > >> > > think we > >> > > > > > > > > should do this > >> > > > > > > > > > > > > > in > >> > > > > > > > > > > > > > > > > 3.0.0. > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > Nick Dimiduk <[email protected]> > >> > > 于2024年9月19日周四 > >> > > > > > > > > 16:46写道： > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > Should we take the opportunity to > >> upgrade > >> > > the > >> > > > > > generic > >> > > > > > > > > > > > > > definitions in > >> > > > > > > > > > > > > > > > > our > >> > > > > > > > > > > > > > > > > > > coprocessor signatures as well? > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > >> > > > >> https://github.com/apache/hbase/blob/edbb145a3f0c6847f444fefdadea5b82d02b0bdb/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java#L101 > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > On Tue, Sep 17, 2024 at 4:41 PM Duo > >> Zhang < > >> > > > > > > > > [email protected]> > >> > > > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > I've done several rounds of API > >> cleanup > >> > > in > >> > > > > > > > > HBASE-24888 and now > >> > > > > > > > > > > > > > for me > >> > > > > > > > > > > > > > > > > > > > I think it is enough for making the > >> new > >> > > major > >> > > > > > > > > release. > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > I've filed HBASE-28844 for aligning > >> the > >> > > commit > >> > > > > > > > > history and jira > >> > > > > > > > > > > > > > > > > > > > issues, it will be a huge project > >> for a > >> > > major > >> > > > > > > > > release and may > >> > > > > > > > > > > > > > take > >> > > > > > > > > > > > > > > > > > > > several weeks. > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > After that, I think we are good to > >> go. > >> > > And I > >> > > > > > will > >> > > > > > > > > also try to > >> > > > > > > > > > > > > > find > >> > > > > > > > > > > > > > > > > > > > some resources to run several > >> rounds of > >> > > ITBLL > >> > > > > > and > >> > > > > > > > > also some > >> > > > > > > > > > > > > > upgrading > >> > > > > > > > > > > > > > > > > > > > tests. Finally, after everything is > >> OK, > >> > > I will > >> > > > > > > > > release > >> > > > > > > > > > > > > > 3.0.0-beta-2, > >> > > > > > > > > > > > > > > > > > > > and then cut branch-3.0 from the > >> > > 3.0.0-beta-2 > >> > > > > > tag > >> > > > > > > > > and make the > >> > > > > > > > > > > > > > 3.0.0 > >> > > > > > > > > > > > > > > > > > > > release soon. > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > Hope I can get all these things done > >> > > within the > >> > > > > > > > year > >> > > > > > > > > 2024. > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > Thanks. > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > -- > >> > > > > > > > *István Tóth* | Sr. Staff Software Engineer > >> > > > > > > > *Email*: [email protected] > >> > > > > > > > cloudera.com <https://www.cloudera.com> > >> > > > > > > > [image: Cloudera] <https://www.cloudera.com/> > >> > > > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> > >> > > [image: > >> > > > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> > >> > > [image: > >> > > > > > Cloudera > >> > > > > > > > on LinkedIn] <https://www.linkedin.com/company/cloudera> > >> > > > > > > > ------------------------------ > >> > > > > > > > ------------------------------ > >> > > > > > > > > >> > > > > > > >> > > > >> > > >> > > >> > -- > >> > *István Tóth* | Sr. Staff Software Engineer > >> > *Email*: [email protected] > >> > cloudera.com <https://www.cloudera.com> > >> > [image: Cloudera] <https://www.cloudera.com/> > >> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > >> > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > >> Cloudera > >> > on LinkedIn] <https://www.linkedin.com/company/cloudera> > >> > ------------------------------ > >> > ------------------------------ > >> > > > > > > -- > > *István Tóth* | Sr. Staff Software Engineer > > *Email*: [email protected] > > cloudera.com <https://www.cloudera.com> > > [image: Cloudera] <https://www.cloudera.com/> > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > > Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> > > ------------------------------ > > ------------------------------ > > > > > -- > *István Tóth* | Sr. Staff Software Engineer > *Email*: [email protected] > cloudera.com <https://www.cloudera.com> > [image: Cloudera] <https://www.cloudera.com/> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera > on LinkedIn] <https://www.linkedin.com/company/cloudera> > ------------------------------ > ------------------------------
