Sorry, I missed your reply eralier, Duo. I still haven't figured out the issue (though at least now I am now able to run those tests from a debugger) No ticket yet, as I'm still not sure which project to use.
To repro this, we need to * install a current HBase 3 snapshot locally from branch-3 head, * check out my WIP Phoenix HBase 3 branch: https://github.com/stoty/phoenix/tree/PHOENIX-7164-2025 * then run a faling test: *mvn clean verify -am -pl phoenix-core -Dit.test=BaseAggregateWithRegionMoves2IT#testGroupByArray -Dhbase.profile=3.0 -Dtest=foo* There are other similar tests failures, missing expected ResultSet rows after aggressively moving around the table regions during the query. Istvan On Thu, Oct 16, 2025 at 3:49 PM 张铎(Duo Zhang) <[email protected]> wrote: > Is there an issue in HBase or Phoenix for tracking this problem? > > Thanks. > > Istvan Toth <[email protected]> 于2025年10月16日周四 11:58写道： > > > > Quick heads up: > > I'm investigating a regression in the Phoenix test suite where scans are > > not reset when the processed region is moved, leading to missing data > from > > the scan. > > > > I am not positive yet that the issue is on the HBase side, will report > here > > (and open a ticket if needed) when I know more. > > > > Istvan > > > > On Sat, Sep 13, 2025 at 2:18 PM 张铎(Duo Zhang) <[email protected]> > wrote: > > > > > Only 4 issues left on 3.0.0-beta-2 now, will wait for one more week to > > > see if we can get these issues done, otherwise will move them out of > > > 3.0.0. > > > > > > And the HBCK2 issue has been fixed. > > > > > > Plan to cut a new hbase-operator-tools release before cutting > 3.0.0-beta-2. > > > > > > Thanks. > > > > > > 张铎(Duo Zhang) <[email protected]> 于2025年8月31日周日 22:57写道： > > > > > > > > I've finished one round of reviewing pending issues for 3.0.0-beta-2. > > > > > > > > There are still some necessary ones we need to get in before the > final > > > > 3.0.0 release, like make HBCK2 work against hbase3. > > > > > > > > Will keep working on them. > > > > > > > > Thanks. > > > > > > > > Nihal Jain <[email protected]> 于2025年6月19日周四 16:06写道： > > > > > > > > > > Yes, from hbase-thirdparty side, release is done and also consumed > in > > > hbase > > > > > branch-2.5 and branch-2.6 and we are good here. > > > > > > > > > > On Wed, 18 Jun 2025 at 8:48 PM, 张铎(Duo Zhang) < > [email protected]> > > > wrote: > > > > > > > > > > > Oh, I think we should cut 2.6.3 and 2.5.12 too? Since we have > already > > > > > > made the new thirdparty release without jetty 12. > > > > > > > > > > > > Thanks. > > > > > > > > > > > > Nihal Jain <[email protected]> 于2025年6月18日周三 01:51写道： > > > > > > > > > > > > > > Ah yes, let me restart work for hbase thirdparty release for > jetty > > > 12 > > > > > > soon. > > > > > > > > > > > > > > On Tue, 17 Jun 2025 at 10:37 PM, Istvan Toth > > > <[email protected] > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > I think we should make sure we're using Jetty 12 in the next > > > beta. ( > > > > > > so we > > > > > > > > need a thirdparty release for Jetty 12) > > > > > > > > We could also review other dependencies, and bump the ones > (if > > > any) > > > > > > that > > > > > > > > were kept on old versions because of the Java 8 requirement. > > > > > > > > > > > > > > > > Istvan > > > > > > > > > > > > > > > > On Tue, Jun 17, 2025 at 11:08 AM 张铎(Duo Zhang) < > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > The good news is that I've finally got a successful 10B > ITBLL > > > run. > > > > > > > > > > > > > > > > > > You can check HBASE-27919 for more details. > > > > > > > > > > > > > > > > > > Next let me cleanup the jira issues and nightly runs. > > > > > > > > > > > > > > > > > > This time I think we can see 3.0.0 out soon. > > > > > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > > > Nihal Jain <[email protected]> 于2025年3月12日周三 03:37写道： > > > > > > > > > > > > > > > > > > > > Hi Duo, > > > > > > > > > > > > > > > > > > > > Have you tried hbase-3.x on a kerberized env? I > encountered > > > > > > HBASE-29144 > > > > > > > > > with kerberos. Posting here to just bring to attention. > > > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > > Nihal > > > > > > > > > > > > > > > > > > > > On 2025/03/08 01:49:12 "张铎(Duo Zhang)" wrote: > > > > > > > > > > > A good news, cluster replication worked fine while > rolling > > > > > > upgrading > > > > > > > > > in my test. > > > > > > > > > > > > > > > > > > > > > > Let me focus on the region replication stuff. After > fixing > > > this > > > > > > > > issue, > > > > > > > > > > > the only blocker is a stable 10B ITBLL run. > > > > > > > > > > > > > > > > > > > > > > Hope we could get 3.0.0 out soon. > > > > > > > > > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > > > > > > > 张铎(Duo Zhang) <[email protected]> 于2025年3月6日周四 > > > 22:55写道： > > > > > > > > > > > > > > > > > > > > > > > > I've finished one round of rolling upgrading test. It > > > worked > > > > > > fine. > > > > > > > > > > > > Please see HBASE-29161. > > > > > > > > > > > > > > > > > > > > > > > > But soon I found a problem when memstore replication > is > > > > > > enabled, > > > > > > > > and > > > > > > > > > > > > filed HBASE-29169. > > > > > > > > > > > > > > > > > > > > > > > > I will test upgrading with cluster replication too, > to > > > see if > > > > > > there > > > > > > > > > > > > are any other issues. > > > > > > > > > > > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > > > > > > > > > Istvan Toth <[email protected]> 于2025年1月20日周一 > 14:24写道： > > > > > > > > > > > > > > > > > > > > > > > > > > For HBASE-28431, only the final step of adding the > > > > > > Hadoop-less > > > > > > > > > assembly is > > > > > > > > > > > > > missing. > > > > > > > > > > > > > > > > > > > > > > > > > > Please review > > > https://github.com/apache/hbase/pull/5766 if > > > > > > you > > > > > > > > > have the > > > > > > > > > > > > > time. > > > > > > > > > > > > > > > > > > > > > > > > > > Istvan > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Dec 31, 2024 at 2:43 PM 张铎(Duo Zhang) < > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > I've finished the first round of issue > alignment, it > > > took > > > > > > > > > several months... > > > > > > > > > > > > > > > > > > > > > > > > > > > > And I've created a release audit tool for 3.0.0 > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache9/hbase-release-audit > > > > > > > > > > > > > > > > > > > > > > > > > > > > where we can have several ignore files, so we do > not > > > need > > > > > > to > > > > > > > > > manually > > > > > > > > > > > > > > check the result every time. > > > > > > > > > > > > > > > > > > > > > > > > > > > > So now, there are still several things to be done > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. Make sure we have all the features and fixes > we > > > want to > > > > > > land > > > > > > > > > on > > > > > > > > > > > > > > branch-3. For example, I think Istvan's assembly > > > cleanup > > > > > > should > > > > > > > > > be > > > > > > > > > > > > > > included. Please report other issues you want to > > > land in > > > > > > final > > > > > > > > > 3.0.0. > > > > > > > > > > > > > > 2. Several stable 10B ITBLL runs. I will try to > see > > > if I > > > > > > can > > > > > > > > get > > > > > > > > > > > > > > enough resources but it will be great if others > > > could help > > > > > > > > here. > > > > > > > > > > > > > > 3. Rolling upgrading test. I plan to do rolling > > > upgrade > > > > > > test > > > > > > > > > with the > > > > > > > > > > > > > > latest 2.6.x and 2.5.x releases. Since we have > big > > > changes > > > > > > on > > > > > > > > > > > > > > replication and region replica replication, I > plan > > > to have > > > > > > > > tests > > > > > > > > > with > > > > > > > > > > > > > > these two features enabled too if possible. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Any other suggestions? > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > > > > > > > > > > > > > 张铎(Duo Zhang) <[email protected]> > 于2024年9月19日周四 > > > > > > 22:35写道： > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > And do we want to apply the same trick to > > > MasterObserver > > > > > > and > > > > > > > > > > > > > > > RegionServerObserver? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > For RegionObserver, we'd better do this because > > > there are > > > > > > > > > already > > > > > > > > > > > > > > > methods in it that use the ObserverContext<? > > > extends > > > > > > > > > > > > > > > RegionCoprocessorEnvironment> declaration, > which > > > were > > > > > > added > > > > > > > > in > > > > > > > > > > > > > > > HBASE-10378. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > For other coprocessor observers, we do not have > > > such > > > > > > methods > > > > > > > > > yet. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Nick Dimiduk <[email protected]> > 于2024年9月19日周四 > > > > > > 22:03写道： > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We do it here or we wait for 4.0. I'm > inclined > > > toward > > > > > > doing > > > > > > > > > it here > > > > > > > > > > > > > > > > because (i think) we expect coprocessor > > > > > > implementations to > > > > > > > > > necessarily > > > > > > > > > > > > > > > > recompile anyway. Though I admit I've not > thought > > > > > > through > > > > > > > > > what > > > > > > > > > > > > > > features are > > > > > > > > > > > > > > > > enabled by this new generic signature. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 19, 2024 at 3:57 PM 张铎(Duo > Zhang) < > > > > > > > > > [email protected]> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Filed HBASE-28862. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It will break old coprocessor > implementations > > > even if > > > > > > > > they > > > > > > > > > do not use > > > > > > > > > > > > > > > > > any deprecated method. But since it does > not > > > change > > > > > > any > > > > > > > > > abilities, I > > > > > > > > > > > > > > > > > think it is OK to include this in a major > > > release as > > > > > > we > > > > > > > > > have already > > > > > > > > > > > > > > > > > done bunch of breaking changes, like > changing > > > the > > > > > > > > protobuf > > > > > > > > > from > > > > > > > > > > > > > > > > > unshaded one to shaded and relocated one... > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 张铎(Duo Zhang) <[email protected]> > > > 于2024年9月19日周四 > > > > > > > > > 17:36写道： > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for pointing this TODO out. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Let me file an issue abort this. For me I > > > think we > > > > > > > > > should do this > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > 3.0.0. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Nick Dimiduk <[email protected]> > > > 于2024年9月19日周四 > > > > > > > > > 16:46写道： > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Should we take the opportunity to > upgrade > > > the > > > > > > generic > > > > > > > > > > > > > > definitions in > > > > > > > > > > > > > > > > > our > > > > > > > > > > > > > > > > > > > coprocessor signatures as well? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/hbase/blob/edbb145a3f0c6847f444fefdadea5b82d02b0bdb/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java#L101 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 17, 2024 at 4:41 PM Duo > Zhang < > > > > > > > > > [email protected]> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I've done several rounds of API > cleanup > > > in > > > > > > > > > HBASE-24888 and now > > > > > > > > > > > > > > for me > > > > > > > > > > > > > > > > > > > > I think it is enough for making the > new > > > major > > > > > > > > > release. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I've filed HBASE-28844 for aligning > the > > > commit > > > > > > > > > history and jira > > > > > > > > > > > > > > > > > > > > issues, it will be a huge project > for a > > > major > > > > > > > > > release and may > > > > > > > > > > > > > > take > > > > > > > > > > > > > > > > > > > > several weeks. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > After that, I think we are good to > go. > > > And I > > > > > > will > > > > > > > > > also try to > > > > > > > > > > > > > > find > > > > > > > > > > > > > > > > > > > > some resources to run several rounds > of > > > ITBLL > > > > > > and > > > > > > > > > also some > > > > > > > > > > > > > > upgrading > > > > > > > > > > > > > > > > > > > > tests. Finally, after everything is > OK, > > > I will > > > > > > > > > release > > > > > > > > > > > > > > 3.0.0-beta-2, > > > > > > > > > > > > > > > > > > > > and then cut branch-3.0 from the > > > 3.0.0-beta-2 > > > > > > tag > > > > > > > > > and make the > > > > > > > > > > > > > > 3.0.0 > > > > > > > > > > > > > > > > > > > > release soon. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hope I can get all these things done > > > within the > > > > > > > > year > > > > > > > > > 2024. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > *István Tóth* | Sr. Staff Software Engineer > > > > > > > > *Email*: [email protected] > > > > > > > > cloudera.com <https://www.cloudera.com> > > > > > > > > [image: Cloudera] <https://www.cloudera.com/> > > > > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> > > > [image: > > > > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> > > > [image: > > > > > > Cloudera > > > > > > > > on LinkedIn] <https://www.linkedin.com/company/cloudera> > > > > > > > > ------------------------------ > > > > > > > > ------------------------------ > > > > > > > > > > > > > > > > > > > > > > > -- > > *István Tóth* | Sr. Staff Software Engineer > > *Email*: [email protected] > > cloudera.com <https://www.cloudera.com> > > [image: Cloudera] <https://www.cloudera.com/> > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > Cloudera > > on LinkedIn] <https://www.linkedin.com/company/cloudera> > > ------------------------------ > > ------------------------------ > -- *István Tóth* | Sr. Staff Software Engineer *Email*: [email protected] cloudera.com <https://www.cloudera.com> [image: Cloudera] <https://www.cloudera.com/> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> ------------------------------ ------------------------------
