Thanks, Istvan, Richard, Josh and Rajeshbabu (and anyone else I missed :-)
) for all your hard work getting master and the ancillary projects into a
releasable state.
An additional task that I think should happen before 5.1 can be released is
getting the indexing code in master up to parity with 4.x. As you may know,
between 4.14 and the upcoming 4.16, a lot of work has been done to build a
new, self-repairing consistent secondary index framework for Phoenix. Most
of that work has been ported to the master branch, but there are still some
significant gaps.
The biggest gap comes from the new indexing framework relying heavily on
Phoenix's SCN "lookback" feature to do point-in-time selects during index
creation and verification. SCN has in the past been an unreliable tool,
because HBase's flush and major compaction code cleans up expired versions
and removes chunks of prior history. To work around this, in 4.16 we've
introduced "max lookback age" in PHOENIX-5645, which allows operators to
configure a moving window during which compactions and flushes will not
purge versions for any history.
PHOENIX-5645 doesn't exist in master, because the coprocessor changes in
HBase 2.0 made implementing the needed compaction hooks impossible.
HBASE-24321, released in HBase 2.3, makes it possible again, though only
for Phoenix builds using the 2.3 profile. (Since it adds to an interface,
HBase compatibility guidelines prevent HBASE-24321 from being backported to
2.1 or 2.2.)
So, for 5.1 I believe we need forward ports for :
PHOENIX-5881 (implementing max lookback age for 2.3) - BLOCKER
PHOENIX-5735 (IndexTool verification distinguishes between inconsistencies
before or after max lookback age) - BLOCKER
PHOENIX-5928 (Simplification / perf improvement for index builds)
PHOENIX-5969 (Bug fix for querying indexes with limit clauses) - BLOCKER
PHOENIX-5951 (Configurable failure logging for past-max-lookback rows)
PHOENIX-6058 (Better behavior on verification when max lookback is disabled
-- needed for HBase 2.1 and 2.2 profiles) - BLOCKER
Kadir, Swaroopa, Abhishek, and Gokcen, please add in any that I missed. :-)
Happy to discuss what is and isn't a blocker.
I plan to do PHOENIX-5881 next week, and the rest depends on it and will
follow afterward.
Thanks,
Geoffrey
On Mon, Aug 3, 2020 at 7:24 AM Istvan Toth <[email protected]> wrote:
Hi!
It's been more than two years since we've released 5.0.0, and almost as
long since Connectors and PQS have been split from the main repo.
I believe that we are now at the point where we've solved, or are close
to
solving the issues that have prevented us from releasing a useful and
relevant 5.1.0 , as well as making an actual releases of PQS and
Connectors
that are usable with both 5.x and 4.x.
The two major blockers that are still open are
- PHOENIX-6010 Create phoenix-thirdparty, and consume guava through it
- PHOENIX-5784 Phoenix-connectors doesn't work with phoenix master
branch
but I hope that we can wrap those up in the next few weeks.
This is going to be a complex process, as we'll have to release new
versions of ALL of our components. To recap, the affected projects, (and
their dependencies):
- phoenix-thirdparty
- tephra
- omid (phoenix-thirdparty)
- phoenix (tehpra ?, omid, phoenix-thirdparty)
- PQS
- Connectors
The 5.1 release is also a point where we can revisit the decision to
support Tephra. We have inherited those projects because of low developer
interest, and it hasn't increased visibly since we've adopted them.
Rajeshbabu and Josh have done some analysis and, as a part of our day
job,
are investing time first with Omid to ensure it's functional with the
rest
of Phoenix in its new home/packaging.
Tephra also carries the technical debt of being dependent on the
discontinued Twill library, which in turn is locked to oid Guava
versions.
In TEPHRA-308 I am implementing the stopgap solution of shading both
away,
so it is not a blocker for 5.1, but concentrating on one library would
probably be a smarter use of the almost non-existent developer time that
goes into maintaining our transactional solution.
I plan to add a profile to build Phoenix without Tephra, thus avoiding
the
problematic dependencies that it has. (Alternatively, the default can be
omitting Tephra, and defining a profile where it is added.)
The effect on 4.x
Short-term, the above releases do not affect 4.x, as it can stay on the
old
omid and tephra dependencies. Having an official release of PQS and
Connectors is a clear win, and Richard Antal is also working on updating
some of the connectors for 4.x.
Mid-term, updating to the new Omid version will bring in the
phoenix-thirdparty dependency to 4.x, and I think it would be smart to
backport the phoenix-thirdparty changes to 4.x as well.
I do not know if there are plans for 4.16 near term. Having 4.x and 5.x
releases that are close feature and bugfix wise would be advantageous in
terms of documenting and communicating them to the users, but it hasn't
stopped either branch from releasing in the past, so releasing 5.1 and
4.16
close together would be a nice-to-have, but not a show stopper.
Also, I have concentrated on the build and dependencies side of Phoenix.
AFAICT there are no major new features going in now that would warrant
delaying the release, I can mostly see the usual fixes and optimizations
these days among the commits.
Thanks for taking the time to read this.
Looking forward to your questions, comments, and opinion.
regards
Istvan
