Hi! Strongly Consistent Global Indexes is indeed the killer new feature in 5.1, and it's important to have it working as well as possible. I was only vaguely aware of the deficiencies in master, thanks for explaining it (and working on fixing them)
Thanks Istvan On Tue, Aug 4, 2020 at 9:22 PM Geoffrey Jacoby <[email protected]> wrote: > Thanks, Istvan, Richard, Josh and Rajeshbabu (and anyone else I missed :-) > ) for all your hard work getting master and the ancillary projects into a > releasable state. > > An additional task that I think should happen before 5.1 can be released is > getting the indexing code in master up to parity with 4.x. As you may know, > between 4.14 and the upcoming 4.16, a lot of work has been done to build a > new, self-repairing consistent secondary index framework for Phoenix. Most > of that work has been ported to the master branch, but there are still some > significant gaps. > > The biggest gap comes from the new indexing framework relying heavily on > Phoenix's SCN "lookback" feature to do point-in-time selects during index > creation and verification. SCN has in the past been an unreliable tool, > because HBase's flush and major compaction code cleans up expired versions > and removes chunks of prior history. To work around this, in 4.16 we've > introduced "max lookback age" in PHOENIX-5645, which allows operators to > configure a moving window during which compactions and flushes will not > purge versions for any history. > > PHOENIX-5645 doesn't exist in master, because the coprocessor changes in > HBase 2.0 made implementing the needed compaction hooks impossible. > HBASE-24321, released in HBase 2.3, makes it possible again, though only > for Phoenix builds using the 2.3 profile. (Since it adds to an interface, > HBase compatibility guidelines prevent HBASE-24321 from being backported to > 2.1 or 2.2.) > > So, for 5.1 I believe we need forward ports for : > PHOENIX-5881 (implementing max lookback age for 2.3) - BLOCKER > PHOENIX-5735 (IndexTool verification distinguishes between inconsistencies > before or after max lookback age) - BLOCKER > PHOENIX-5928 (Simplification / perf improvement for index builds) > PHOENIX-5969 (Bug fix for querying indexes with limit clauses) - BLOCKER > PHOENIX-5951 (Configurable failure logging for past-max-lookback rows) > PHOENIX-6058 (Better behavior on verification when max lookback is disabled > -- needed for HBase 2.1 and 2.2 profiles) - BLOCKER > > Kadir, Swaroopa, Abhishek, and Gokcen, please add in any that I missed. :-) > Happy to discuss what is and isn't a blocker. > > I plan to do PHOENIX-5881 next week, and the rest depends on it and will > follow afterward. > > Thanks, > > Geoffrey > > > On Mon, Aug 3, 2020 at 7:24 AM Istvan Toth <[email protected]> wrote: > > > Hi! > > > > It's been more than two years since we've released 5.0.0, and almost as > > long since Connectors and PQS have been split from the main repo. > > > > I believe that we are now at the point where we've solved, or are close > to > > solving the issues that have prevented us from releasing a useful and > > relevant 5.1.0 , as well as making an actual releases of PQS and > Connectors > > that are usable with both 5.x and 4.x. > > > > The two major blockers that are still open are > > > > - PHOENIX-6010 Create phoenix-thirdparty, and consume guava through it > > - PHOENIX-5784 Phoenix-connectors doesn't work with phoenix master > > branch > > > > but I hope that we can wrap those up in the next few weeks. > > > > This is going to be a complex process, as we'll have to release new > > versions of ALL of our components. To recap, the affected projects, (and > > their dependencies): > > > > - phoenix-thirdparty > > - tephra > > - omid (phoenix-thirdparty) > > - phoenix (tehpra ?, omid, phoenix-thirdparty) > > - PQS > > - Connectors > > > > The 5.1 release is also a point where we can revisit the decision to > > support Tephra. We have inherited those projects because of low developer > > interest, and it hasn't increased visibly since we've adopted them. > > Rajeshbabu and Josh have done some analysis and, as a part of our day > job, > > are investing time first with Omid to ensure it's functional with the > rest > > of Phoenix in its new home/packaging. > > > > Tephra also carries the technical debt of being dependent on the > > discontinued Twill library, which in turn is locked to oid Guava > versions. > > In TEPHRA-308 I am implementing the stopgap solution of shading both > away, > > so it is not a blocker for 5.1, but concentrating on one library would > > probably be a smarter use of the almost non-existent developer time that > > goes into maintaining our transactional solution. > > > > I plan to add a profile to build Phoenix without Tephra, thus avoiding > the > > problematic dependencies that it has. (Alternatively, the default can be > > omitting Tephra, and defining a profile where it is added.) > > > > > > The effect on 4.x > > > > Short-term, the above releases do not affect 4.x, as it can stay on the > old > > omid and tephra dependencies. Having an official release of PQS and > > Connectors is a clear win, and Richard Antal is also working on updating > > some of the connectors for 4.x. > > > > Mid-term, updating to the new Omid version will bring in the > > phoenix-thirdparty dependency to 4.x, and I think it would be smart to > > backport the phoenix-thirdparty changes to 4.x as well. > > > > I do not know if there are plans for 4.16 near term. Having 4.x and 5.x > > releases that are close feature and bugfix wise would be advantageous in > > terms of documenting and communicating them to the users, but it hasn't > > stopped either branch from releasing in the past, so releasing 5.1 and > 4.16 > > close together would be a nice-to-have, but not a show stopper. > > > > Also, I have concentrated on the build and dependencies side of Phoenix. > > AFAICT there are no major new features going in now that would warrant > > delaying the release, I can mostly see the usual fixes and optimizations > > these days among the commits. > > > > Thanks for taking the time to read this. > > > > Looking forward to your questions, comments, and opinion. > > > > regards > > > > Istvan > > > -- *István Tóth* | Staff Software Engineer [email protected] <https://www.cloudera.com> [image: Cloudera] <https://www.cloudera.com/> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> <https://www.cloudera.com/> ------------------------------
