Thanks, Istvan, Richard, Josh and Rajeshbabu (and anyone else I missed :-) ) for all your hard work getting master and the ancillary projects into a releasable state.
An additional task that I think should happen before 5.1 can be released is getting the indexing code in master up to parity with 4.x. As you may know, between 4.14 and the upcoming 4.16, a lot of work has been done to build a new, self-repairing consistent secondary index framework for Phoenix. Most of that work has been ported to the master branch, but there are still some significant gaps. The biggest gap comes from the new indexing framework relying heavily on Phoenix's SCN "lookback" feature to do point-in-time selects during index creation and verification. SCN has in the past been an unreliable tool, because HBase's flush and major compaction code cleans up expired versions and removes chunks of prior history. To work around this, in 4.16 we've introduced "max lookback age" in PHOENIX-5645, which allows operators to configure a moving window during which compactions and flushes will not purge versions for any history. PHOENIX-5645 doesn't exist in master, because the coprocessor changes in HBase 2.0 made implementing the needed compaction hooks impossible. HBASE-24321, released in HBase 2.3, makes it possible again, though only for Phoenix builds using the 2.3 profile. (Since it adds to an interface, HBase compatibility guidelines prevent HBASE-24321 from being backported to 2.1 or 2.2.) So, for 5.1 I believe we need forward ports for : PHOENIX-5881 (implementing max lookback age for 2.3) - BLOCKER PHOENIX-5735 (IndexTool verification distinguishes between inconsistencies before or after max lookback age) - BLOCKER PHOENIX-5928 (Simplification / perf improvement for index builds) PHOENIX-5969 (Bug fix for querying indexes with limit clauses) - BLOCKER PHOENIX-5951 (Configurable failure logging for past-max-lookback rows) PHOENIX-6058 (Better behavior on verification when max lookback is disabled -- needed for HBase 2.1 and 2.2 profiles) - BLOCKER Kadir, Swaroopa, Abhishek, and Gokcen, please add in any that I missed. :-) Happy to discuss what is and isn't a blocker. I plan to do PHOENIX-5881 next week, and the rest depends on it and will follow afterward. Thanks, Geoffrey On Mon, Aug 3, 2020 at 7:24 AM Istvan Toth <[email protected]> wrote: > Hi! > > It's been more than two years since we've released 5.0.0, and almost as > long since Connectors and PQS have been split from the main repo. > > I believe that we are now at the point where we've solved, or are close to > solving the issues that have prevented us from releasing a useful and > relevant 5.1.0 , as well as making an actual releases of PQS and Connectors > that are usable with both 5.x and 4.x. > > The two major blockers that are still open are > > - PHOENIX-6010 Create phoenix-thirdparty, and consume guava through it > - PHOENIX-5784 Phoenix-connectors doesn't work with phoenix master > branch > > but I hope that we can wrap those up in the next few weeks. > > This is going to be a complex process, as we'll have to release new > versions of ALL of our components. To recap, the affected projects, (and > their dependencies): > > - phoenix-thirdparty > - tephra > - omid (phoenix-thirdparty) > - phoenix (tehpra ?, omid, phoenix-thirdparty) > - PQS > - Connectors > > The 5.1 release is also a point where we can revisit the decision to > support Tephra. We have inherited those projects because of low developer > interest, and it hasn't increased visibly since we've adopted them. > Rajeshbabu and Josh have done some analysis and, as a part of our day job, > are investing time first with Omid to ensure it's functional with the rest > of Phoenix in its new home/packaging. > > Tephra also carries the technical debt of being dependent on the > discontinued Twill library, which in turn is locked to oid Guava versions. > In TEPHRA-308 I am implementing the stopgap solution of shading both away, > so it is not a blocker for 5.1, but concentrating on one library would > probably be a smarter use of the almost non-existent developer time that > goes into maintaining our transactional solution. > > I plan to add a profile to build Phoenix without Tephra, thus avoiding the > problematic dependencies that it has. (Alternatively, the default can be > omitting Tephra, and defining a profile where it is added.) > > > The effect on 4.x > > Short-term, the above releases do not affect 4.x, as it can stay on the old > omid and tephra dependencies. Having an official release of PQS and > Connectors is a clear win, and Richard Antal is also working on updating > some of the connectors for 4.x. > > Mid-term, updating to the new Omid version will bring in the > phoenix-thirdparty dependency to 4.x, and I think it would be smart to > backport the phoenix-thirdparty changes to 4.x as well. > > I do not know if there are plans for 4.16 near term. Having 4.x and 5.x > releases that are close feature and bugfix wise would be advantageous in > terms of documenting and communicating them to the users, but it hasn't > stopped either branch from releasing in the past, so releasing 5.1 and 4.16 > close together would be a nice-to-have, but not a show stopper. > > Also, I have concentrated on the build and dependencies side of Phoenix. > AFAICT there are no major new features going in now that would warrant > delaying the release, I can mostly see the usual fixes and optimizations > these days among the commits. > > Thanks for taking the time to read this. > > Looking forward to your questions, comments, and opinion. > > regards > > Istvan >
