We're making progress. I can see that Viraj has just landed PHOENIX-7601, and Rajeshbabu has released Omid 1.1.1. Thank you!
At the moment, the following outstanding issues are on my radar: https://issues.apache.org/jira/browse/PHOENIX-7191 https://issues.apache.org/jira/browse/PHOENIX-7193 These are bugs in my non-ZK registry implementation, which were found during HBase 3 work. I have some PRs up, but they may not be complete. I will push for reviews once I have the HBase 3 tests passing, and possibly updated them based on that. We also have a number of very flakey tests, see: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-mulitbranch/job/master/test_results_analyzer/ On Tue, Jan 30, 2024 at 7:09 AM Istvan Toth <st...@cloudera.com> wrote: > As Viraj wrote, those are just plans. > If HBase 3 won't be released by the time the other features are ready, > then it won't make it into 5.3. > If other major features are ready by that time, then they will be > included. (though we are not aware of any now) > > As for the new major version, in the past Phoenix didn't have a > compatibility module system, > so a new branch was required, which didn't support older HBases. Also, > the API changes between HBase 1.x and 2.x were much larger, > The HBase 2 and 3 API are pretty similar, apart from the removal of > deprecated 1.x APIs. (and the protobuf/protocol thing, which requires a > rather ugly hack). > > I will start the discussion on how we can add HBase 3 support as soon as I > have a working POC patch. > > We could call 5.3 6.0 instead, after all Phoenix isn't using a strict > semantic versioning, but then 6.0 would also support HBase 2. > If we do not come to a consensus on the version name, we can always have a > vote on it. > > I think that the main motivation is that the community wants to maintain > as few branches as possible. > > Istvan > > On Mon, Jan 29, 2024 at 10:02 PM Stephen Jiang <syuanjiang...@gmail.com> > wrote: > >> I am not sure how close HBase 3.0 is. Even if it is only less than one >> year away, the adoption would be low at the beginning. I don't think 5.3 >> should wait for that. And traditionally, Phoenix would have a major >> release to support the HBase major release (4.x for HBase 1.x and 5.x for >> HBase 2.x), in this case, we are talking about Phoenix 6.0 for HBase 3.0. >> >> Maybe we should adopt the HBase release model: master branch for next >> major >> release (6.0) and branch-5.x branch for next 5.x minor release and >> branch-5.2 for 5.2 minor release. >> >> Thanks >> Stephen >> >> >> >> On Thu, Jan 25, 2024 at 11:50 PM Viraj Jasani <vjas...@apache.org> wrote: >> >> > Sounds good. >> > >> > Planned major changes for 5.3.0: >> > >> > 1. JSON support. >> > 2. HBase 3.0 support. >> > 3. CDC feature (leveraging uncovered global index framework and JSON >> > support). >> > >> > >> > On Wed, Jan 24, 2024 at 8:22 PM Kadir Ozdemir >> > <kozde...@salesforce.com.invalid> wrote: >> > >> > > I suggest including another major change for 5.3, Phoenix CDC, >> > > https://issues.apache.org/jira/browse/PHOENIX-7001. The PR for it >> will >> > be >> > > posted soon. >> > > >> > > On Thu, Jan 25, 2024 at 5:35 AM Viraj Jasani <vjas...@apache.org> >> wrote: >> > > >> > > > Thanks Istvan! I agree with your points. It’s really been a while we >> > are >> > > > talking about releasing 5.2.0 and yet due to bandwidth issues, >> unable >> > to >> > > do >> > > > so. >> > > > >> > > > I agree to getting these fixes out, cut 5.2 and start the release >> work >> > > and >> > > > in the meantime I also need to prepare 5.1 backport. >> > > > >> > > > For 5.3, let's plan HBase 3.0 support and JSON as major changes. >> > > > >> > > > >> > > > On Mon, Jan 22, 2024 at 9:17 PM Istvan Toth <st...@apache.org> >> wrote: >> > > > >> > > > > Hi! >> > > > > >> > > > > In my opinion cutting 5.2 now only makes sense IF we DO NOT plan >> to >> > > > release >> > > > > the outstanding big features (like JSON) in 5.2. , otherwise it's >> > just >> > > > > extra work to maintain more branches. >> > > > > >> > > > > Having said that, releasing a 5.2 and 5.1.4 with the data >> integrity >> > > fixes >> > > > > real soon, and then releasing 5.3 in a few months with JSON, and >> any >> > > > other >> > > > > outstanding big features >> > > > > that are close to being finished (and HBase 3.0 support, if it's >> > ready >> > > by >> > > > > then) would not be a bad idea. >> > > > > >> > > > > On the CLDR side the only outstanding big feature which could >> impact >> > > > > Viraj's integrity work is HBase 3.0 support, and even that is only >> > > > because >> > > > > it may require some larger refactors of existing code, not >> because it >> > > > would >> > > > > change the actual behaviour or algorithms. >> > > > > >> > > > > Phoenix used to have several minor releases per year, the current >> > state >> > > > of >> > > > > extreme longevity of 5.1 and several big new features being added >> to >> > it >> > > > > (like uncovered indexes) is not ideal. >> > > > > Releasing 5.2 and 5.3 relatively close together could be a return >> to >> > a >> > > > > quicker cadence for minor releases, which could also help with the >> > > public >> > > > > image and adoption of Phoenix. >> > > > > >> > > > > We were talking about releasing 5.2 at least a year ago, and I >> have >> > > > started >> > > > > working on that then, but then emergencies have come up at >> $dayjob, >> > > and I >> > > > > could not see that through. >> > > > > (So I am in part responsible for the lack of minor releases) >> > > > > >> > > > > regards >> > > > > Istvan >> > > > > >> > > > > >> > > > > On Mon, Jan 22, 2024 at 11:35 PM Viraj Jasani <vjas...@apache.org >> > >> > > > wrote: >> > > > > >> > > > > > Sorry for the late reply. >> > > > > > >> > > > > > > Do you think cutting 5.2.0 now makes sense? >> > > > > > >> > > > > > No problem with that. I can cut 5.2 branch by the end of this >> week >> > or >> > > > at >> > > > > > the start of next week. >> > > > > > >> > > > > > If there is any very big change or feature ready for merge to >> > master >> > > > > branch >> > > > > > with PR approvals already in place, please do let me know so >> that I >> > > can >> > > > > > help collaborate on how best we can get it merged without >> impacting >> > > 5.2 >> > > > > > release if required. My main motivation was for any big change >> to >> > go >> > > > > > through newly introduced tests so that we know that anything >> > > additional >> > > > > is >> > > > > > not broken, and also to prioritize for upcoming 5.2.0 and 5.1.4 >> > > > releases. >> > > > > > Moreover, there are several PRs getting merged on the master >> > branch, >> > > we >> > > > > can >> > > > > > continue that as long as they are not very big changes, which >> might >> > > > > require >> > > > > > significant time to understand any correlation with data >> integrity >> > > > > issues. >> > > > > > >> > > > > > The PR is also ready for review with some additional cases fixed >> > last >> > > > > week: >> > > > > > >> > > > >> > > >> > >> https://urldefense.com/v3/__https://github.com/apache/phoenix/pull/1736__;!!DCbAVzZNrAf4!D4OVjUp2EWW2BqhGnBxsapDX_AHsibRphIpoFBWfgRsd3dsAikrFLo6PGxdTzGbSXJJ2fJ0j9mcz3asXMXo$ >> > > > > > >> > > > > > Depending on the review bandwidth, I am hopeful we should be >> good >> > to >> > > > land >> > > > > > them sooner. >> > > > > > >> > > > > > >> > > > > > On Wed, Jan 17, 2024 at 11:31 AM Rushabh Shah >> > > > > > <rushabh.s...@salesforce.com.invalid> wrote: >> > > > > > >> > > > > > > Thank you Viraj for initiating this thread. >> > > > > > > >> > > > > > > > Given the critical nature of these issues, I would like to >> > > propose >> > > > > that >> > > > > > > we >> > > > > > > treat this as a high priority for the upcoming 5.2.0 release, >> and >> > > not >> > > > > > > include any other feature or big change to master branch >> until we >> > > > merge >> > > > > > > this. >> > > > > > > >> > > > > > > Do you think cutting 5.2.0 now makes sense? This will enable >> > other >> > > > > > > developers to merge features into master branch (5.3.0) and >> you >> > can >> > > > > take >> > > > > > > some more time to make sure we cover all the corner cases for >> the >> > > > data >> > > > > > > integrity issues that you uncovered. >> > > > > > > >> > > > > > > >> > > > > > > On Fri, Jan 5, 2024 at 6:38 PM Viraj Jasani < >> vjas...@apache.org> >> > > > > wrote: >> > > > > > > >> > > > > > > > Sounds good, thanks Rajeshbabu. I will try to get the first >> PR >> > > out >> > > > > next >> > > > > > > > week and while reviews happen in parallel, will try to get >> 5.1 >> > PR >> > > > > soon. >> > > > > > > > >> > > > > > > > >> > > > > > > > On Wed, Jan 3, 2024 at 8:49 PM rajeshb...@apache.org < >> > > > > > > > chrajeshbab...@gmail.com> wrote: >> > > > > > > > >> > > > > > > > > Hi Viraj, >> > > > > > > > > >> > > > > > > > > Would be better to include the changes in 5.1.4 as in any >> > way >> > > it >> > > > > > will >> > > > > > > > take >> > > > > > > > > at least 3-4 days to complete the omid release. >> > > > > > > > > >> > > > > > > > > Thanks, >> > > > > > > > > Rajeshbabu. >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > On Thu, Jan 4, 2024 at 5:06 AM Viraj Jasani < >> > > vjas...@apache.org> >> > > > > > > wrote: >> > > > > > > > > >> > > > > > > > > > Thank you Kadir and Geoffrey for your replies!! >> > > > > > > > > > >> > > > > > > > > > > How does this affect 5.1.4, which is also listed as a >> Fix >> > > > > Version >> > > > > > > for >> > > > > > > > > > > PHOENIX-7106? >> > > > > > > > > > >> > > > > > > > > > Yes, it also needs to be ported to 5.1. Once the master >> PR >> > is >> > > > up >> > > > > > for >> > > > > > > > > final >> > > > > > > > > > review, I would start working on the backport PR. >> > > > > > > > > > We just need some more additional testing to ensure old >> > > client >> > > > > > (e.g. >> > > > > > > > > 5.1.3) >> > > > > > > > > > is compatible with the new server with the changes. >> > > > > > > > > > >> > > > > > > > > > Hence, yes it is now a blocker for upcoming 5.1.4 as >> well >> > > since >> > > > > > 5.1.4 >> > > > > > > > RC >> > > > > > > > > > preparation is still pending (while Omid release is in >> > > > progress). >> > > > > > > > > > Otherwise if 5.1.4 was ready for release, I would have >> > > proposed >> > > > > > > > immediate >> > > > > > > > > > 5.1.5 release to include the changes proposed with >> > > > PHOENIX-7106. >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > On Wed, Jan 3, 2024 at 3:08 PM Geoffrey Jacoby < >> > > > > gjac...@apache.org >> > > > > > > >> > > > > > > > > wrote: >> > > > > > > > > > >> > > > > > > > > > > I agree that data integrity issues are a higher >> priority >> > > than >> > > > > > > feature >> > > > > > > > > > > development, so I also support the decision. The fact >> > that >> > > > > > several >> > > > > > > of >> > > > > > > > > the >> > > > > > > > > > > major remaining 5.2 features are currently being >> > developed >> > > in >> > > > > > > > > > long-running >> > > > > > > > > > > feature branches also helps, as work can continue >> there >> > at >> > > > the >> > > > > > cost >> > > > > > > > of >> > > > > > > > > a >> > > > > > > > > > > rebase later. >> > > > > > > > > > > >> > > > > > > > > > > How does this affect 5.1.4, which is also listed as a >> Fix >> > > > > Version >> > > > > > > for >> > > > > > > > > > > PHOENIX-7106? From the bug description it also sounds >> > like >> > > > > 5.1.3 >> > > > > > > and >> > > > > > > > > the >> > > > > > > > > > > forthcoming .4 are affected, since we have server-side >> > > paging >> > > > > in >> > > > > > > 5.1. >> > > > > > > > > > (Feel >> > > > > > > > > > > free to move that to a separate thread if you feel it >> > > should >> > > > > be a >> > > > > > > > > > separate >> > > > > > > > > > > discussion.) Should this be a blocker for releasing >> > 5.1.4? >> > > > > > > > > > > >> > > > > > > > > > > Geoffrey >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > On Wed, Jan 3, 2024 at 5:06 PM Kadir Ozdemir < >> > > > > > > > > > > ka...@gsuite.cloud.apache.org> >> > > > > > > > > > > wrote: >> > > > > > > > > > > >> > > > > > > > > > > > Being a database, Phoenix has to make sure that the >> > data >> > > > > stays >> > > > > > on >> > > > > > > > > disk >> > > > > > > > > > > > intact and its queries return correct data. In this >> > case, >> > > > > > Phoenix >> > > > > > > > > fails >> > > > > > > > > > > to >> > > > > > > > > > > > return correct data for some queries if their scans >> > > > > experience >> > > > > > > > region >> > > > > > > > > > > > movement. Now that we know these data integrity >> issues >> > > and >> > > > > how >> > > > > > to >> > > > > > > > > > > reproduce >> > > > > > > > > > > > them, fixing them should be our first priority. So, >> I >> > > fully >> > > > > > > support >> > > > > > > > > > this >> > > > > > > > > > > > proposal. >> > > > > > > > > > > > >> > > > > > > > > > > > On Wed, Jan 3, 2024 at 10:58 PM Viraj Jasani < >> > > > > > vjas...@apache.org >> > > > > > > > >> > > > > > > > > > wrote: >> > > > > > > > > > > > >> > > > > > > > > > > > > Hello, >> > > > > > > > > > > > > >> > > > > > > > > > > > > I would like to bring PHOENIX-7106 >> > > > > > > > > > > > > < >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/PHOENIX-7106__;!!DCbAVzZNrAf4!FZG5sv55IC1NqItQLY7GKWgUG2Do0gSta01gOiSdd36Dx3XHGtQx4M3c9visVXIt9DctPQzS-orob9vhzrCfVA$ >> > > > > > > > > to everyone's >> > > > > > > > > > > > > attention here and brief about the data integrity >> > > issues >> > > > > that >> > > > > > > we >> > > > > > > > > have >> > > > > > > > > > > in >> > > > > > > > > > > > > various coprocessors. Majority of the issues are >> > > related >> > > > to >> > > > > > the >> > > > > > > > > fact >> > > > > > > > > > > that >> > > > > > > > > > > > > we do not return valid rowkey for certain >> queries. If >> > > any >> > > > > > > region >> > > > > > > > > > moves >> > > > > > > > > > > in >> > > > > > > > > > > > > the middle of the scan, the HBase client relies on >> > the >> > > > last >> > > > > > > > > returned >> > > > > > > > > > > > rowkey >> > > > > > > > > > > > > and accordingly changes the scan boundaries while >> the >> > > > > scanner >> > > > > > > is >> > > > > > > > > > > getting >> > > > > > > > > > > > > reset to continue the scan operation. If the >> region >> > > does >> > > > > not >> > > > > > > > move, >> > > > > > > > > > scan >> > > > > > > > > > > > is >> > > > > > > > > > > > > not expected to return invalid data, however if >> the >> > > > region >> > > > > > > moves >> > > > > > > > in >> > > > > > > > > > the >> > > > > > > > > > > > > middle of ongoing scan operation, scan would >> return >> > > > > > > > > invalid/incorrect >> > > > > > > > > > > > data >> > > > > > > > > > > > > causing data integrity issues. >> > > > > > > > > > > > > >> > > > > > > > > > > > > Given the critical nature of these issues, I would >> > like >> > > > to >> > > > > > > > propose >> > > > > > > > > > that >> > > > > > > > > > > > we >> > > > > > > > > > > > > treat this as a high priority for the upcoming >> 5.2.0 >> > > > > release, >> > > > > > > and >> > > > > > > > > not >> > > > > > > > > > > > > include any other feature or big change to master >> > > branch >> > > > > > until >> > > > > > > we >> > > > > > > > > > merge >> > > > > > > > > > > > > this. The PR is still not ready as additional >> changes >> > > are >> > > > > > still >> > > > > > > > in >> > > > > > > > > my >> > > > > > > > > > > > > local, requiring rebase with the current master. >> > > > > > > > > > > > > >> > > > > > > > > > > > > I would get back to this discuss thread as soon as >> > the >> > > PR >> > > > > and >> > > > > > > the >> > > > > > > > > doc >> > > > > > > > > > > are >> > > > > > > > > > > > > updated with the latest findings so far. The >> changes >> > > > > include >> > > > > > > many >> > > > > > > > > of >> > > > > > > > > > > our >> > > > > > > > > > > > > coproc scanner implementations and hence it would >> > > require >> > > > > > > > > significant >> > > > > > > > > > > > > review as well. >> > > > > > > > > > > > > It would be great if we can hold on to merging any >> > > > feature >> > > > > or >> > > > > > > big >> > > > > > > > > > > change >> > > > > > > > > > > > to >> > > > > > > > > > > > > master branch until this gets in so as to not >> > > complicate >> > > > > > > > > > > > merging/rebasing. >> > > > > > > > > > > > > Once this is merged to the master branch, I would >> > like >> > > to >> > > > > cut >> > > > > > > 5.2 >> > > > > > > > > > > branch >> > > > > > > > > > > > > from master and we can move forward with 5.2.0 >> > release. >> > > > > > > > > > > > > >> > > > > > > > > > > > > Please let me know if this looks good or if you >> have >> > > any >> > > > > > other >> > > > > > > > high >> > > > > > > > > > > > > priority work for 5.2.0. >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> > > > -- > *István Tóth* | Sr. Staff Software Engineer > *Email*: st...@cloudera.com > cloudera.com <https://www.cloudera.com> > [image: Cloudera] <https://www.cloudera.com/> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> > ------------------------------ > ------------------------------ >
