Yes, they basically make the non-ZK registries unusable. (at least the connectionless problems should be fixed.)
I hope to have the final fix for those sometime next week. Also have we looked at potential CVE issues on master recently ? I think we should also look at the most flakey tests I linked above, and fix them or at least make sure that they are test issues and not real bugs. Istvan On Fri, Feb 9, 2024 at 6:55 PM Viraj Jasani <vjas...@apache.org> wrote: > Thanks Istvan. > I would like to cut 5.2 branch from master. Do you see non-ZK registry for > MapReduce jobs as blocker for 5.2.0? > > > On Fri, Feb 9, 2024 at 12:24 AM Istvan Toth <st...@apache.org> wrote: > > > ParallelPhoenixConnectionFailureTest.testExecuteQueryChainFailure also > > fails too often, especially when the test host is slow and/or the load is > > high. > > On my fast laptop, I can semi-reliably break it by running > > mvn clean verify -am -pl phoenix-core -DnumForkedUT=20 > > > > On Fri, Feb 9, 2024 at 9:19 AM Istvan Toth <st...@apache.org> wrote: > > > > > We're making progress. > > > I can see that Viraj has just landed PHOENIX-7601, and Rajeshbabu has > > > released Omid 1.1.1. > > > Thank you! > > > > > > At the moment, the following outstanding issues are on my radar: > > > > > > https://issues.apache.org/jira/browse/PHOENIX-7191 > > > https://issues.apache.org/jira/browse/PHOENIX-7193 > > > > > > These are bugs in my non-ZK registry implementation, which were found > > > during HBase 3 work. > > > I have some PRs up, but they may not be complete. I will push for > reviews > > > once I have the HBase 3 tests passing, and possibly updated them based > on > > > that. > > > > > > We also have a number of very flakey tests, see: > > > > > > > > > > > > https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-mulitbranch/job/master/test_results_analyzer/ > > > > > > > > > > > > On Tue, Jan 30, 2024 at 7:09 AM Istvan Toth <st...@cloudera.com> > wrote: > > > > > >> As Viraj wrote, those are just plans. > > >> If HBase 3 won't be released by the time the other features are ready, > > >> then it won't make it into 5.3. > > >> If other major features are ready by that time, then they will be > > >> included. (though we are not aware of any now) > > >> > > >> As for the new major version, in the past Phoenix didn't have a > > >> compatibility module system, > > >> so a new branch was required, which didn't support older HBases. > Also, > > >> the API changes between HBase 1.x and 2.x were much larger, > > >> The HBase 2 and 3 API are pretty similar, apart from the removal of > > >> deprecated 1.x APIs. (and the protobuf/protocol thing, which requires > a > > >> rather ugly hack). > > >> > > >> I will start the discussion on how we can add HBase 3 support as soon > as > > >> I have a working POC patch. > > >> > > >> We could call 5.3 6.0 instead, after all Phoenix isn't using a strict > > >> semantic versioning, but then 6.0 would also support HBase 2. > > >> If we do not come to a consensus on the version name, we can always > have > > >> a vote on it. > > >> > > >> I think that the main motivation is that the community wants to > maintain > > >> as few branches as possible. > > >> > > >> Istvan > > >> > > >> On Mon, Jan 29, 2024 at 10:02 PM Stephen Jiang < > syuanjiang...@gmail.com > > > > > >> wrote: > > >> > > >>> I am not sure how close HBase 3.0 is. Even if it is only less than > one > > >>> year away, the adoption would be low at the beginning. I don't think > > 5.3 > > >>> should wait for that. And traditionally, Phoenix would have a major > > >>> release to support the HBase major release (4.x for HBase 1.x and 5.x > > for > > >>> HBase 2.x), in this case, we are talking about Phoenix 6.0 for HBase > > 3.0. > > >>> > > >>> Maybe we should adopt the HBase release model: master branch for next > > >>> major > > >>> release (6.0) and branch-5.x branch for next 5.x minor release and > > >>> branch-5.2 for 5.2 minor release. > > >>> > > >>> Thanks > > >>> Stephen > > >>> > > >>> > > >>> > > >>> On Thu, Jan 25, 2024 at 11:50 PM Viraj Jasani <vjas...@apache.org> > > >>> wrote: > > >>> > > >>> > Sounds good. > > >>> > > > >>> > Planned major changes for 5.3.0: > > >>> > > > >>> > 1. JSON support. > > >>> > 2. HBase 3.0 support. > > >>> > 3. CDC feature (leveraging uncovered global index framework and > JSON > > >>> > support). > > >>> > > > >>> > > > >>> > On Wed, Jan 24, 2024 at 8:22 PM Kadir Ozdemir > > >>> > <kozde...@salesforce.com.invalid> wrote: > > >>> > > > >>> > > I suggest including another major change for 5.3, Phoenix CDC, > > >>> > > https://issues.apache.org/jira/browse/PHOENIX-7001. The PR for > it > > >>> will > > >>> > be > > >>> > > posted soon. > > >>> > > > > >>> > > On Thu, Jan 25, 2024 at 5:35 AM Viraj Jasani <vjas...@apache.org > > > > >>> wrote: > > >>> > > > > >>> > > > Thanks Istvan! I agree with your points. It’s really been a > while > > >>> we > > >>> > are > > >>> > > > talking about releasing 5.2.0 and yet due to bandwidth issues, > > >>> unable > > >>> > to > > >>> > > do > > >>> > > > so. > > >>> > > > > > >>> > > > I agree to getting these fixes out, cut 5.2 and start the > release > > >>> work > > >>> > > and > > >>> > > > in the meantime I also need to prepare 5.1 backport. > > >>> > > > > > >>> > > > For 5.3, let's plan HBase 3.0 support and JSON as major > changes. > > >>> > > > > > >>> > > > > > >>> > > > On Mon, Jan 22, 2024 at 9:17 PM Istvan Toth <st...@apache.org> > > >>> wrote: > > >>> > > > > > >>> > > > > Hi! > > >>> > > > > > > >>> > > > > In my opinion cutting 5.2 now only makes sense IF we DO NOT > > plan > > >>> to > > >>> > > > release > > >>> > > > > the outstanding big features (like JSON) in 5.2. , otherwise > > it's > > >>> > just > > >>> > > > > extra work to maintain more branches. > > >>> > > > > > > >>> > > > > Having said that, releasing a 5.2 and 5.1.4 with the data > > >>> integrity > > >>> > > fixes > > >>> > > > > real soon, and then releasing 5.3 in a few months with JSON, > > and > > >>> any > > >>> > > > other > > >>> > > > > outstanding big features > > >>> > > > > that are close to being finished (and HBase 3.0 support, if > > it's > > >>> > ready > > >>> > > by > > >>> > > > > then) would not be a bad idea. > > >>> > > > > > > >>> > > > > On the CLDR side the only outstanding big feature which could > > >>> impact > > >>> > > > > Viraj's integrity work is HBase 3.0 support, and even that is > > >>> only > > >>> > > > because > > >>> > > > > it may require some larger refactors of existing code, not > > >>> because it > > >>> > > > would > > >>> > > > > change the actual behaviour or algorithms. > > >>> > > > > > > >>> > > > > Phoenix used to have several minor releases per year, the > > current > > >>> > state > > >>> > > > of > > >>> > > > > extreme longevity of 5.1 and several big new features being > > >>> added to > > >>> > it > > >>> > > > > (like uncovered indexes) is not ideal. > > >>> > > > > Releasing 5.2 and 5.3 relatively close together could be a > > >>> return to > > >>> > a > > >>> > > > > quicker cadence for minor releases, which could also help > with > > >>> the > > >>> > > public > > >>> > > > > image and adoption of Phoenix. > > >>> > > > > > > >>> > > > > We were talking about releasing 5.2 at least a year ago, and > I > > >>> have > > >>> > > > started > > >>> > > > > working on that then, but then emergencies have come up at > > >>> $dayjob, > > >>> > > and I > > >>> > > > > could not see that through. > > >>> > > > > (So I am in part responsible for the lack of minor releases) > > >>> > > > > > > >>> > > > > regards > > >>> > > > > Istvan > > >>> > > > > > > >>> > > > > > > >>> > > > > On Mon, Jan 22, 2024 at 11:35 PM Viraj Jasani < > > >>> vjas...@apache.org> > > >>> > > > wrote: > > >>> > > > > > > >>> > > > > > Sorry for the late reply. > > >>> > > > > > > > >>> > > > > > > Do you think cutting 5.2.0 now makes sense? > > >>> > > > > > > > >>> > > > > > No problem with that. I can cut 5.2 branch by the end of > this > > >>> week > > >>> > or > > >>> > > > at > > >>> > > > > > the start of next week. > > >>> > > > > > > > >>> > > > > > If there is any very big change or feature ready for merge > to > > >>> > master > > >>> > > > > branch > > >>> > > > > > with PR approvals already in place, please do let me know > so > > >>> that I > > >>> > > can > > >>> > > > > > help collaborate on how best we can get it merged without > > >>> impacting > > >>> > > 5.2 > > >>> > > > > > release if required. My main motivation was for any big > > change > > >>> to > > >>> > go > > >>> > > > > > through newly introduced tests so that we know that > anything > > >>> > > additional > > >>> > > > > is > > >>> > > > > > not broken, and also to prioritize for upcoming 5.2.0 and > > 5.1.4 > > >>> > > > releases. > > >>> > > > > > Moreover, there are several PRs getting merged on the > master > > >>> > branch, > > >>> > > we > > >>> > > > > can > > >>> > > > > > continue that as long as they are not very big changes, > which > > >>> might > > >>> > > > > require > > >>> > > > > > significant time to understand any correlation with data > > >>> integrity > > >>> > > > > issues. > > >>> > > > > > > > >>> > > > > > The PR is also ready for review with some additional cases > > >>> fixed > > >>> > last > > >>> > > > > week: > > >>> > > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > https://urldefense.com/v3/__https://github.com/apache/phoenix/pull/1736__;!!DCbAVzZNrAf4!D4OVjUp2EWW2BqhGnBxsapDX_AHsibRphIpoFBWfgRsd3dsAikrFLo6PGxdTzGbSXJJ2fJ0j9mcz3asXMXo$ > > >>> > > > > > > > >>> > > > > > Depending on the review bandwidth, I am hopeful we should > be > > >>> good > > >>> > to > > >>> > > > land > > >>> > > > > > them sooner. > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > On Wed, Jan 17, 2024 at 11:31 AM Rushabh Shah > > >>> > > > > > <rushabh.s...@salesforce.com.invalid> wrote: > > >>> > > > > > > > >>> > > > > > > Thank you Viraj for initiating this thread. > > >>> > > > > > > > > >>> > > > > > > > Given the critical nature of these issues, I would like > > to > > >>> > > propose > > >>> > > > > that > > >>> > > > > > > we > > >>> > > > > > > treat this as a high priority for the upcoming 5.2.0 > > >>> release, and > > >>> > > not > > >>> > > > > > > include any other feature or big change to master branch > > >>> until we > > >>> > > > merge > > >>> > > > > > > this. > > >>> > > > > > > > > >>> > > > > > > Do you think cutting 5.2.0 now makes sense? This will > > enable > > >>> > other > > >>> > > > > > > developers to merge features into master branch (5.3.0) > and > > >>> you > > >>> > can > > >>> > > > > take > > >>> > > > > > > some more time to make sure we cover all the corner cases > > >>> for the > > >>> > > > data > > >>> > > > > > > integrity issues that you uncovered. > > >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > On Fri, Jan 5, 2024 at 6:38 PM Viraj Jasani < > > >>> vjas...@apache.org> > > >>> > > > > wrote: > > >>> > > > > > > > > >>> > > > > > > > Sounds good, thanks Rajeshbabu. I will try to get the > > >>> first PR > > >>> > > out > > >>> > > > > next > > >>> > > > > > > > week and while reviews happen in parallel, will try to > > get > > >>> 5.1 > > >>> > PR > > >>> > > > > soon. > > >>> > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > On Wed, Jan 3, 2024 at 8:49 PM rajeshb...@apache.org < > > >>> > > > > > > > chrajeshbab...@gmail.com> wrote: > > >>> > > > > > > > > > >>> > > > > > > > > Hi Viraj, > > >>> > > > > > > > > > > >>> > > > > > > > > Would be better to include the changes in 5.1.4 as > in > > >>> any > > >>> > way > > >>> > > it > > >>> > > > > > will > > >>> > > > > > > > take > > >>> > > > > > > > > at least 3-4 days to complete the omid release. > > >>> > > > > > > > > > > >>> > > > > > > > > Thanks, > > >>> > > > > > > > > Rajeshbabu. > > >>> > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > On Thu, Jan 4, 2024 at 5:06 AM Viraj Jasani < > > >>> > > vjas...@apache.org> > > >>> > > > > > > wrote: > > >>> > > > > > > > > > > >>> > > > > > > > > > Thank you Kadir and Geoffrey for your replies!! > > >>> > > > > > > > > > > > >>> > > > > > > > > > > How does this affect 5.1.4, which is also listed > as > > >>> a Fix > > >>> > > > > Version > > >>> > > > > > > for > > >>> > > > > > > > > > > PHOENIX-7106? > > >>> > > > > > > > > > > > >>> > > > > > > > > > Yes, it also needs to be ported to 5.1. Once the > > >>> master PR > > >>> > is > > >>> > > > up > > >>> > > > > > for > > >>> > > > > > > > > final > > >>> > > > > > > > > > review, I would start working on the backport PR. > > >>> > > > > > > > > > We just need some more additional testing to ensure > > old > > >>> > > client > > >>> > > > > > (e.g. > > >>> > > > > > > > > 5.1.3) > > >>> > > > > > > > > > is compatible with the new server with the changes. > > >>> > > > > > > > > > > > >>> > > > > > > > > > Hence, yes it is now a blocker for upcoming 5.1.4 > as > > >>> well > > >>> > > since > > >>> > > > > > 5.1.4 > > >>> > > > > > > > RC > > >>> > > > > > > > > > preparation is still pending (while Omid release is > > in > > >>> > > > progress). > > >>> > > > > > > > > > Otherwise if 5.1.4 was ready for release, I would > > have > > >>> > > proposed > > >>> > > > > > > > immediate > > >>> > > > > > > > > > 5.1.5 release to include the changes proposed with > > >>> > > > PHOENIX-7106. > > >>> > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > On Wed, Jan 3, 2024 at 3:08 PM Geoffrey Jacoby < > > >>> > > > > gjac...@apache.org > > >>> > > > > > > > > >>> > > > > > > > > wrote: > > >>> > > > > > > > > > > > >>> > > > > > > > > > > I agree that data integrity issues are a higher > > >>> priority > > >>> > > than > > >>> > > > > > > feature > > >>> > > > > > > > > > > development, so I also support the decision. The > > fact > > >>> > that > > >>> > > > > > several > > >>> > > > > > > of > > >>> > > > > > > > > the > > >>> > > > > > > > > > > major remaining 5.2 features are currently being > > >>> > developed > > >>> > > in > > >>> > > > > > > > > > long-running > > >>> > > > > > > > > > > feature branches also helps, as work can continue > > >>> there > > >>> > at > > >>> > > > the > > >>> > > > > > cost > > >>> > > > > > > > of > > >>> > > > > > > > > a > > >>> > > > > > > > > > > rebase later. > > >>> > > > > > > > > > > > > >>> > > > > > > > > > > How does this affect 5.1.4, which is also listed > as > > >>> a Fix > > >>> > > > > Version > > >>> > > > > > > for > > >>> > > > > > > > > > > PHOENIX-7106? From the bug description it also > > sounds > > >>> > like > > >>> > > > > 5.1.3 > > >>> > > > > > > and > > >>> > > > > > > > > the > > >>> > > > > > > > > > > forthcoming .4 are affected, since we have > > >>> server-side > > >>> > > paging > > >>> > > > > in > > >>> > > > > > > 5.1. > > >>> > > > > > > > > > (Feel > > >>> > > > > > > > > > > free to move that to a separate thread if you > feel > > it > > >>> > > should > > >>> > > > > be a > > >>> > > > > > > > > > separate > > >>> > > > > > > > > > > discussion.) Should this be a blocker for > releasing > > >>> > 5.1.4? > > >>> > > > > > > > > > > > > >>> > > > > > > > > > > Geoffrey > > >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > > On Wed, Jan 3, 2024 at 5:06 PM Kadir Ozdemir < > > >>> > > > > > > > > > > ka...@gsuite.cloud.apache.org> > > >>> > > > > > > > > > > wrote: > > >>> > > > > > > > > > > > > >>> > > > > > > > > > > > Being a database, Phoenix has to make sure that > > the > > >>> > data > > >>> > > > > stays > > >>> > > > > > on > > >>> > > > > > > > > disk > > >>> > > > > > > > > > > > intact and its queries return correct data. In > > this > > >>> > case, > > >>> > > > > > Phoenix > > >>> > > > > > > > > fails > > >>> > > > > > > > > > > to > > >>> > > > > > > > > > > > return correct data for some queries if their > > scans > > >>> > > > > experience > > >>> > > > > > > > region > > >>> > > > > > > > > > > > movement. Now that we know these data integrity > > >>> issues > > >>> > > and > > >>> > > > > how > > >>> > > > > > to > > >>> > > > > > > > > > > reproduce > > >>> > > > > > > > > > > > them, fixing them should be our first priority. > > >>> So, I > > >>> > > fully > > >>> > > > > > > support > > >>> > > > > > > > > > this > > >>> > > > > > > > > > > > proposal. > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > On Wed, Jan 3, 2024 at 10:58 PM Viraj Jasani < > > >>> > > > > > vjas...@apache.org > > >>> > > > > > > > > > >>> > > > > > > > > > wrote: > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > Hello, > > >>> > > > > > > > > > > > > > > >>> > > > > > > > > > > > > I would like to bring PHOENIX-7106 > > >>> > > > > > > > > > > > > < > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > https://urldefense.com/v3/__https://issues.apache.org/jira/browse/PHOENIX-7106__;!!DCbAVzZNrAf4!FZG5sv55IC1NqItQLY7GKWgUG2Do0gSta01gOiSdd36Dx3XHGtQx4M3c9visVXIt9DctPQzS-orob9vhzrCfVA$ > > >>> > > > > > > > > to everyone's > > >>> > > > > > > > > > > > > attention here and brief about the data > > integrity > > >>> > > issues > > >>> > > > > that > > >>> > > > > > > we > > >>> > > > > > > > > have > > >>> > > > > > > > > > > in > > >>> > > > > > > > > > > > > various coprocessors. Majority of the issues > > are > > >>> > > related > > >>> > > > to > > >>> > > > > > the > > >>> > > > > > > > > fact > > >>> > > > > > > > > > > that > > >>> > > > > > > > > > > > > we do not return valid rowkey for certain > > >>> queries. If > > >>> > > any > > >>> > > > > > > region > > >>> > > > > > > > > > moves > > >>> > > > > > > > > > > in > > >>> > > > > > > > > > > > > the middle of the scan, the HBase client > relies > > >>> on > > >>> > the > > >>> > > > last > > >>> > > > > > > > > returned > > >>> > > > > > > > > > > > rowkey > > >>> > > > > > > > > > > > > and accordingly changes the scan boundaries > > >>> while the > > >>> > > > > scanner > > >>> > > > > > > is > > >>> > > > > > > > > > > getting > > >>> > > > > > > > > > > > > reset to continue the scan operation. If the > > >>> region > > >>> > > does > > >>> > > > > not > > >>> > > > > > > > move, > > >>> > > > > > > > > > scan > > >>> > > > > > > > > > > > is > > >>> > > > > > > > > > > > > not expected to return invalid data, however > if > > >>> the > > >>> > > > region > > >>> > > > > > > moves > > >>> > > > > > > > in > > >>> > > > > > > > > > the > > >>> > > > > > > > > > > > > middle of ongoing scan operation, scan would > > >>> return > > >>> > > > > > > > > invalid/incorrect > > >>> > > > > > > > > > > > data > > >>> > > > > > > > > > > > > causing data integrity issues. > > >>> > > > > > > > > > > > > > > >>> > > > > > > > > > > > > Given the critical nature of these issues, I > > >>> would > > >>> > like > > >>> > > > to > > >>> > > > > > > > propose > > >>> > > > > > > > > > that > > >>> > > > > > > > > > > > we > > >>> > > > > > > > > > > > > treat this as a high priority for the > upcoming > > >>> 5.2.0 > > >>> > > > > release, > > >>> > > > > > > and > > >>> > > > > > > > > not > > >>> > > > > > > > > > > > > include any other feature or big change to > > master > > >>> > > branch > > >>> > > > > > until > > >>> > > > > > > we > > >>> > > > > > > > > > merge > > >>> > > > > > > > > > > > > this. The PR is still not ready as additional > > >>> changes > > >>> > > are > > >>> > > > > > still > > >>> > > > > > > > in > > >>> > > > > > > > > my > > >>> > > > > > > > > > > > > local, requiring rebase with the current > > master. > > >>> > > > > > > > > > > > > > > >>> > > > > > > > > > > > > I would get back to this discuss thread as > soon > > >>> as > > >>> > the > > >>> > > PR > > >>> > > > > and > > >>> > > > > > > the > > >>> > > > > > > > > doc > > >>> > > > > > > > > > > are > > >>> > > > > > > > > > > > > updated with the latest findings so far. The > > >>> changes > > >>> > > > > include > > >>> > > > > > > many > > >>> > > > > > > > > of > > >>> > > > > > > > > > > our > > >>> > > > > > > > > > > > > coproc scanner implementations and hence it > > would > > >>> > > require > > >>> > > > > > > > > significant > > >>> > > > > > > > > > > > > review as well. > > >>> > > > > > > > > > > > > It would be great if we can hold on to > merging > > >>> any > > >>> > > > feature > > >>> > > > > or > > >>> > > > > > > big > > >>> > > > > > > > > > > change > > >>> > > > > > > > > > > > to > > >>> > > > > > > > > > > > > master branch until this gets in so as to not > > >>> > > complicate > > >>> > > > > > > > > > > > merging/rebasing. > > >>> > > > > > > > > > > > > Once this is merged to the master branch, I > > would > > >>> > like > > >>> > > to > > >>> > > > > cut > > >>> > > > > > > 5.2 > > >>> > > > > > > > > > > branch > > >>> > > > > > > > > > > > > from master and we can move forward with > 5.2.0 > > >>> > release. > > >>> > > > > > > > > > > > > > > >>> > > > > > > > > > > > > Please let me know if this looks good or if > you > > >>> have > > >>> > > any > > >>> > > > > > other > > >>> > > > > > > > high > > >>> > > > > > > > > > > > > priority work for 5.2.0. > > >>> > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > >> > > >> > > >> -- > > >> *István Tóth* | Sr. Staff Software Engineer > > >> *Email*: st...@cloudera.com > > >> cloudera.com <https://www.cloudera.com> > > >> [image: Cloudera] <https://www.cloudera.com/> > > >> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > > >> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > > >> Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> > > >> ------------------------------ > > >> ------------------------------ > > >> > > > > > >
