Some updates here, there is no blocker issue for 3.0.0-beta-2 release now, but the junit5 upgrading is in its last stage, where we want to migrate all the tests in hbase-server, so I plan to finish the migration before cutting the final release.
I will also start to run several rounds of 10B ITBLL to check the stability of branch-3. Thanks. 张铎(Duo Zhang) <[email protected]> 于2026年1月22日周四 19:15写道： > > Thanks Istvan for the detailed analysis! > > For HBASE-29834, I've already put up a PR to fix the problem. See > > https://github.com/apache/hbase/pull/7660 > > And on HBASE-29843, I do not fully understand the problem and need > more context about the usage on Phoenix side. I think we should be > able to fix it. > > Thanks. > > Istvan Toth <[email protected]> 于2026年1月22日周四 14:33写道： > > > > This seems to have been caused by forward synchronous RPC changes between > > Hbase 2 and 3 (where HBase 2 simulates non-inclusive start keys by > > adding a \00 byte and setting them as inclusive, while HBase 3 handles them > > natively. > > This is something that we can handle on the Phoenix side. > > > > On the other hand, I've found a major and a minor breaking change in HBase > > 3: > > > > The big problem is HBASE-29843 > > <https://issues.apache.org/jira/browse/HBASE-29843> > > HBaseRPCController.setPriority(final TableName tn) is never called on > > branch-3+ > > Phoenix relies on setting it its own HBaseRPCController to set custom > > priorities, but HBase 3 has mostly abandoned using HBaseRPCController, > > leaving no > > working mechanism for Phoenix to use. > > HBase needs to restore using it to maintain compatibility. (or provide some > > alternative method) > > > > The smaller problem is HBASE-29834 > > <https://issues.apache.org/jira/browse/HBASE-29834> MILLIS_BETWEEN_NEXTS > > metric is not updated on branch-3+ > > This does not break Phoenix, it just happens to be detected by a Phoenix > > test. > > > > Istvan > > > > > > On Mon, Nov 3, 2025 at 10:22 AM Istvan Toth <[email protected]> wrote: > > > > > Sorry, I missed your reply eralier, Duo. > > > > > > I still haven't figured out the issue (though at least now I am now able > > > to run those tests from a debugger) > > > No ticket yet, as I'm still not sure which project to use. > > > > > > To repro this, we need to > > > * install a current HBase 3 snapshot locally from branch-3 head, > > > * check out my WIP Phoenix HBase 3 branch: > > > https://github.com/stoty/phoenix/tree/PHOENIX-7164-2025 > > > * then run a faling test: > > > > > > *mvn clean verify -am -pl phoenix-core > > > -Dit.test=BaseAggregateWithRegionMoves2IT#testGroupByArray > > > -Dhbase.profile=3.0 -Dtest=foo* > > > There are other similar tests failures, missing expected ResultSet rows > > > after aggressively moving around the table regions during the query. > > > > > > Istvan > > > > > > On Thu, Oct 16, 2025 at 3:49 PM 张铎(Duo Zhang) <[email protected]> > > > wrote: > > > > > >> Is there an issue in HBase or Phoenix for tracking this problem? > > >> > > >> Thanks. > > >> > > >> Istvan Toth <[email protected]> 于2025年10月16日周四 11:58写道： > > >> > > > >> > Quick heads up: > > >> > I'm investigating a regression in the Phoenix test suite where scans > > >> > are > > >> > not reset when the processed region is moved, leading to missing data > > >> from > > >> > the scan. > > >> > > > >> > I am not positive yet that the issue is on the HBase side, will report > > >> here > > >> > (and open a ticket if needed) when I know more. > > >> > > > >> > Istvan > > >> > > > >> > On Sat, Sep 13, 2025 at 2:18 PM 张铎(Duo Zhang) <[email protected]> > > >> wrote: > > >> > > > >> > > Only 4 issues left on 3.0.0-beta-2 now, will wait for one more week > > >> > > to > > >> > > see if we can get these issues done, otherwise will move them out of > > >> > > 3.0.0. > > >> > > > > >> > > And the HBCK2 issue has been fixed. > > >> > > > > >> > > Plan to cut a new hbase-operator-tools release before cutting > > >> 3.0.0-beta-2. > > >> > > > > >> > > Thanks. > > >> > > > > >> > > 张铎(Duo Zhang) <[email protected]> 于2025年8月31日周日 22:57写道： > > >> > > > > > >> > > > I've finished one round of reviewing pending issues for > > >> 3.0.0-beta-2. > > >> > > > > > >> > > > There are still some necessary ones we need to get in before the > > >> final > > >> > > > 3.0.0 release, like make HBCK2 work against hbase3. > > >> > > > > > >> > > > Will keep working on them. > > >> > > > > > >> > > > Thanks. > > >> > > > > > >> > > > Nihal Jain <[email protected]> 于2025年6月19日周四 16:06写道： > > >> > > > > > > >> > > > > Yes, from hbase-thirdparty side, release is done and also > > >> consumed in > > >> > > hbase > > >> > > > > branch-2.5 and branch-2.6 and we are good here. > > >> > > > > > > >> > > > > On Wed, 18 Jun 2025 at 8:48 PM, 张铎(Duo Zhang) < > > >> [email protected]> > > >> > > wrote: > > >> > > > > > > >> > > > > > Oh, I think we should cut 2.6.3 and 2.5.12 too? Since we have > > >> already > > >> > > > > > made the new thirdparty release without jetty 12. > > >> > > > > > > > >> > > > > > Thanks. > > >> > > > > > > > >> > > > > > Nihal Jain <[email protected]> 于2025年6月18日周三 01:51写道： > > >> > > > > > > > > >> > > > > > > Ah yes, let me restart work for hbase thirdparty release for > > >> jetty > > >> > > 12 > > >> > > > > > soon. > > >> > > > > > > > > >> > > > > > > On Tue, 17 Jun 2025 at 10:37 PM, Istvan Toth > > >> > > <[email protected] > > >> > > > > > > > > >> > > > > > > wrote: > > >> > > > > > > > > >> > > > > > > > I think we should make sure we're using Jetty 12 in the > > >> > > > > > > > next > > >> > > beta. ( > > >> > > > > > so we > > >> > > > > > > > need a thirdparty release for Jetty 12) > > >> > > > > > > > We could also review other dependencies, and bump the ones > > >> (if > > >> > > any) > > >> > > > > > that > > >> > > > > > > > were kept on old versions because of the Java 8 > > >> > > > > > > > requirement. > > >> > > > > > > > > > >> > > > > > > > Istvan > > >> > > > > > > > > > >> > > > > > > > On Tue, Jun 17, 2025 at 11:08 AM 张铎(Duo Zhang) < > > >> > > [email protected]> > > >> > > > > > > > wrote: > > >> > > > > > > > > > >> > > > > > > > > The good news is that I've finally got a successful 10B > > >> ITBLL > > >> > > run. > > >> > > > > > > > > > > >> > > > > > > > > You can check HBASE-27919 for more details. > > >> > > > > > > > > > > >> > > > > > > > > Next let me cleanup the jira issues and nightly runs. > > >> > > > > > > > > > > >> > > > > > > > > This time I think we can see 3.0.0 out soon. > > >> > > > > > > > > > > >> > > > > > > > > Thanks. > > >> > > > > > > > > > > >> > > > > > > > > Nihal Jain <[email protected]> 于2025年3月12日周三 03:37写道： > > >> > > > > > > > > > > > >> > > > > > > > > > Hi Duo, > > >> > > > > > > > > > > > >> > > > > > > > > > Have you tried hbase-3.x on a kerberized env? I > > >> encountered > > >> > > > > > HBASE-29144 > > >> > > > > > > > > with kerberos. Posting here to just bring to attention. > > >> > > > > > > > > > > > >> > > > > > > > > > Regards > > >> > > > > > > > > > Nihal > > >> > > > > > > > > > > > >> > > > > > > > > > On 2025/03/08 01:49:12 "张铎(Duo Zhang)" wrote: > > >> > > > > > > > > > > A good news, cluster replication worked fine while > > >> rolling > > >> > > > > > upgrading > > >> > > > > > > > > in my test. > > >> > > > > > > > > > > > > >> > > > > > > > > > > Let me focus on the region replication stuff. After > > >> fixing > > >> > > this > > >> > > > > > > > issue, > > >> > > > > > > > > > > the only blocker is a stable 10B ITBLL run. > > >> > > > > > > > > > > > > >> > > > > > > > > > > Hope we could get 3.0.0 out soon. > > >> > > > > > > > > > > > > >> > > > > > > > > > > Thanks. > > >> > > > > > > > > > > > > >> > > > > > > > > > > 张铎(Duo Zhang) <[email protected]> 于2025年3月6日周四 > > >> > > 22:55写道： > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > I've finished one round of rolling upgrading test. > > >> It > > >> > > worked > > >> > > > > > fine. > > >> > > > > > > > > > > > Please see HBASE-29161. > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > But soon I found a problem when memstore > > >> replication is > > >> > > > > > enabled, > > >> > > > > > > > and > > >> > > > > > > > > > > > filed HBASE-29169. > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > I will test upgrading with cluster replication too, > > >> to > > >> > > see if > > >> > > > > > there > > >> > > > > > > > > > > > are any other issues. > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > Thanks. > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > Istvan Toth <[email protected]> 于2025年1月20日周一 > > >> 14:24写道： > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > For HBASE-28431, only the final step of adding > > >> > > > > > > > > > > > > the > > >> > > > > > Hadoop-less > > >> > > > > > > > > assembly is > > >> > > > > > > > > > > > > missing. > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > Please review > > >> > > https://github.com/apache/hbase/pull/5766 if > > >> > > > > > you > > >> > > > > > > > > have the > > >> > > > > > > > > > > > > time. > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > Istvan > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > On Tue, Dec 31, 2024 at 2:43 PM 张铎(Duo Zhang) < > > >> > > > > > > > > [email protected]> wrote: > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > I've finished the first round of issue > > >> alignment, it > > >> > > took > > >> > > > > > > > > several months... > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > And I've created a release audit tool for 3.0.0 > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > https://github.com/apache9/hbase-release-audit > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > where we can have several ignore files, so we > > >> do not > > >> > > need > > >> > > > > > to > > >> > > > > > > > > manually > > >> > > > > > > > > > > > > > check the result every time. > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > So now, there are still several things to be > > >> done > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > 1. Make sure we have all the features and fixes > > >> we > > >> > > want to > > >> > > > > > land > > >> > > > > > > > > on > > >> > > > > > > > > > > > > > branch-3. For example, I think Istvan's > > >> > > > > > > > > > > > > > assembly > > >> > > cleanup > > >> > > > > > should > > >> > > > > > > > > be > > >> > > > > > > > > > > > > > included. Please report other issues you want > > >> > > > > > > > > > > > > > to > > >> > > land in > > >> > > > > > final > > >> > > > > > > > > 3.0.0. > > >> > > > > > > > > > > > > > 2. Several stable 10B ITBLL runs. I will try to > > >> see > > >> > > if I > > >> > > > > > can > > >> > > > > > > > get > > >> > > > > > > > > > > > > > enough resources but it will be great if others > > >> > > could help > > >> > > > > > > > here. > > >> > > > > > > > > > > > > > 3. Rolling upgrading test. I plan to do rolling > > >> > > upgrade > > >> > > > > > test > > >> > > > > > > > > with the > > >> > > > > > > > > > > > > > latest 2.6.x and 2.5.x releases. Since we have > > >> big > > >> > > changes > > >> > > > > > on > > >> > > > > > > > > > > > > > replication and region replica replication, I > > >> plan > > >> > > to have > > >> > > > > > > > tests > > >> > > > > > > > > with > > >> > > > > > > > > > > > > > these two features enabled too if possible. > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > Any other suggestions? > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > Thanks. > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > 张铎(Duo Zhang) <[email protected]> > > >> 于2024年9月19日周四 > > >> > > > > > 22:35写道： > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > And do we want to apply the same trick to > > >> > > MasterObserver > > >> > > > > > and > > >> > > > > > > > > > > > > > > RegionServerObserver? > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > For RegionObserver, we'd better do this > > >> because > > >> > > there are > > >> > > > > > > > > already > > >> > > > > > > > > > > > > > > methods in it that use the ObserverContext<? > > >> > > extends > > >> > > > > > > > > > > > > > > RegionCoprocessorEnvironment> declaration, > > >> which > > >> > > were > > >> > > > > > added > > >> > > > > > > > in > > >> > > > > > > > > > > > > > > HBASE-10378. > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > For other coprocessor observers, we do not > > >> have > > >> > > such > > >> > > > > > methods > > >> > > > > > > > > yet. > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Nick Dimiduk <[email protected]> > > >> 于2024年9月19日周四 > > >> > > > > > 22:03写道： > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > We do it here or we wait for 4.0. I'm > > >> inclined > > >> > > toward > > >> > > > > > doing > > >> > > > > > > > > it here > > >> > > > > > > > > > > > > > > > because (i think) we expect coprocessor > > >> > > > > > implementations to > > >> > > > > > > > > necessarily > > >> > > > > > > > > > > > > > > > recompile anyway. Though I admit I've not > > >> thought > > >> > > > > > through > > >> > > > > > > > > what > > >> > > > > > > > > > > > > > features are > > >> > > > > > > > > > > > > > > > enabled by this new generic signature. > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > On Thu, Sep 19, 2024 at 3:57 PM 张铎(Duo > > >> Zhang) < > > >> > > > > > > > > [email protected]> > > >> > > > > > > > > > > > > > wrote: > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > Filed HBASE-28862. > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > It will break old coprocessor > > >> implementations > > >> > > even if > > >> > > > > > > > they > > >> > > > > > > > > do not use > > >> > > > > > > > > > > > > > > > > any deprecated method. But since it does > > >> not > > >> > > change > > >> > > > > > any > > >> > > > > > > > > abilities, I > > >> > > > > > > > > > > > > > > > > think it is OK to include this in a major > > >> > > release as > > >> > > > > > we > > >> > > > > > > > > have already > > >> > > > > > > > > > > > > > > > > done bunch of breaking changes, like > > >> changing > > >> > > the > > >> > > > > > > > protobuf > > >> > > > > > > > > from > > >> > > > > > > > > > > > > > > > > unshaded one to shaded and relocated > > >> one... > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > Thanks. > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > 张铎(Duo Zhang) <[email protected]> > > >> > > 于2024年9月19日周四 > > >> > > > > > > > > 17:36写道： > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > Thanks for pointing this TODO out. > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > Let me file an issue abort this. For me > > >> I > > >> > > think we > > >> > > > > > > > > should do this > > >> > > > > > > > > > > > > > in > > >> > > > > > > > > > > > > > > > > 3.0.0. > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > Nick Dimiduk <[email protected]> > > >> > > 于2024年9月19日周四 > > >> > > > > > > > > 16:46写道： > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > Should we take the opportunity to > > >> upgrade > > >> > > the > > >> > > > > > generic > > >> > > > > > > > > > > > > > definitions in > > >> > > > > > > > > > > > > > > > > our > > >> > > > > > > > > > > > > > > > > > > coprocessor signatures as well? > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > >> > > > > >> https://github.com/apache/hbase/blob/edbb145a3f0c6847f444fefdadea5b82d02b0bdb/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java#L101 > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > On Tue, Sep 17, 2024 at 4:41 PM Duo > > >> Zhang < > > >> > > > > > > > > [email protected]> > > >> > > > > > > > > > > > > > wrote: > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > I've done several rounds of API > > >> cleanup > > >> > > in > > >> > > > > > > > > HBASE-24888 and now > > >> > > > > > > > > > > > > > for me > > >> > > > > > > > > > > > > > > > > > > > I think it is enough for making the > > >> new > > >> > > major > > >> > > > > > > > > release. > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > I've filed HBASE-28844 for aligning > > >> the > > >> > > commit > > >> > > > > > > > > history and jira > > >> > > > > > > > > > > > > > > > > > > > issues, it will be a huge project > > >> for a > > >> > > major > > >> > > > > > > > > release and may > > >> > > > > > > > > > > > > > take > > >> > > > > > > > > > > > > > > > > > > > several weeks. > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > After that, I think we are good to > > >> go. > > >> > > And I > > >> > > > > > will > > >> > > > > > > > > also try to > > >> > > > > > > > > > > > > > find > > >> > > > > > > > > > > > > > > > > > > > some resources to run several > > >> rounds of > > >> > > ITBLL > > >> > > > > > and > > >> > > > > > > > > also some > > >> > > > > > > > > > > > > > upgrading > > >> > > > > > > > > > > > > > > > > > > > tests. Finally, after everything is > > >> OK, > > >> > > I will > > >> > > > > > > > > release > > >> > > > > > > > > > > > > > 3.0.0-beta-2, > > >> > > > > > > > > > > > > > > > > > > > and then cut branch-3.0 from the > > >> > > 3.0.0-beta-2 > > >> > > > > > tag > > >> > > > > > > > > and make the > > >> > > > > > > > > > > > > > 3.0.0 > > >> > > > > > > > > > > > > > > > > > > > release soon. > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > Hope I can get all these things > > >> > > > > > > > > > > > > > > > > > > > done > > >> > > within the > > >> > > > > > > > year > > >> > > > > > > > > 2024. > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > Thanks. > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > -- > > >> > > > > > > > *István Tóth* | Sr. Staff Software Engineer > > >> > > > > > > > *Email*: [email protected] > > >> > > > > > > > cloudera.com <https://www.cloudera.com> > > >> > > > > > > > [image: Cloudera] <https://www.cloudera.com/> > > >> > > > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> > > >> > > [image: > > >> > > > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> > > >> > > [image: > > >> > > > > > Cloudera > > >> > > > > > > > on LinkedIn] <https://www.linkedin.com/company/cloudera> > > >> > > > > > > > ------------------------------ > > >> > > > > > > > ------------------------------ > > >> > > > > > > > > > >> > > > > > > > >> > > > > >> > > > >> > > > >> > -- > > >> > *István Tóth* | Sr. Staff Software Engineer > > >> > *Email*: [email protected] > > >> > cloudera.com <https://www.cloudera.com> > > >> > [image: Cloudera] <https://www.cloudera.com/> > > >> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > > >> > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > > >> Cloudera > > >> > on LinkedIn] <https://www.linkedin.com/company/cloudera> > > >> > ------------------------------ > > >> > ------------------------------ > > >> > > > > > > > > > -- > > > *István Tóth* | Sr. Staff Software Engineer > > > *Email*: [email protected] > > > cloudera.com <https://www.cloudera.com> > > > [image: Cloudera] <https://www.cloudera.com/> > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > > > Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> > > > ------------------------------ > > > ------------------------------ > > > > > > > > > -- > > *István Tóth* | Sr. Staff Software Engineer > > *Email*: [email protected] > > cloudera.com <https://www.cloudera.com> > > [image: Cloudera] <https://www.cloudera.com/> > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera > > on LinkedIn] <https://www.linkedin.com/company/cloudera> > > ------------------------------ > > ------------------------------
