Hi all,
I propose that we start work toward Phoenix 5.4.0. As such, I
volunteer myself as your RM for the 5.4.0 release and will start
the tasks required to complete it.
Please give your feedback on the plan, and on the questions toward
the end of this mail.
SCOPE
5.4.0 contains everything that has landed on master so far
and including these additional in-flight features:
- EXPLAIN improvements (PHOENIX-7876) -- committed to master
- V2 WHERE optimizer (PHOENIX-6791) -- PR #2428 is approved.
- Consistent Failover (PHOENIX-7562) -- in development on the
PHOENIX-7562-feature-new branch.
- Vector indexes, Phase 1 and Phase 2 (PHOENIX-7998 and
PHOENIX-8009, umbrella PHOENIX-8008) -- Both are in review.
Master also has TRUNCATE TABLE (PHOENIX-300), and numerous other
changes that are not on 5.3, committed over the intervening time.
The V2 WHERE optimizer is off by default in 5.4.0. Users will enable
it per their discretion with phoenix.where.optimizer.v2.enabled=true.
Consistent Failover and Vector Indexes are also opt-in features that
do not have an impact unless configured and enabled.
We will cut the 5.4 branch only after all of the features above are
merged to master. For this reason, the date of the cut depends on the
readiness of the features, with the possible exception of Consistent
Failover, which has 21 open subtasks. Stabilization of 5.4 branch can
begin with most everything else in place and Consistent Failover can
come in at any point as a backport of the merge commit to master
branch.
MINIMUM JAVA VERSION POLICY
5.4.0 as proposed changes the minimum supported Java version policy:
- Minimum: Java 11. 5.4.0 will not build or run on Java 8.
- Recommended: Java 17.
- Suggested for vector indexes: Java 21.
Phoenix ships multi-release jars. On Java 21, the JVM loads vector
kernels that use the Panama Vector API. On Java 11 and 17, it loads
portable kernels.
Although this is a significant change for users, it is unlikely users
remain on Java 8 for a myriad of technical and compliance related
reasons. The release notes will state the policy clearly. Users who
must stay on Java 8 can continue with the 5.3 release line.
PLAN
Phase 1. Land the features on master. PHOENIX-6791 merges first.
PHOENIX-7998 and PHOENIX-8009 merge in sequence. PHOENIX-8009
also moves the build and CI to Java 11. Consistent Failover
merges last.
Phase 2. Cut the 5.4 branch. Master moves to 5.5.0-SNAPSHOT. We
triage the unresolved 5.4.0 issues in JIRA, and keep only the
release blockers.
Phase 3. Stabilize. The work includes these items:
- Fix or quarantine flaky tests.
- Harden each new feature.
- Test compatibility and upgrade from 5.2 and 5.3.
- Build and test each HBase profile on JDK 11, 17, and 21.
- Review licenses for the new dependencies.
- Update the release tooling to JDK 17.
- Check the downstream projects.
- Write the release notes and documentation.
Phase 4. Release candidates and vote.
The Critical issues PHOENIX-7279 and PHOENIX-7596 currently block
releasing.
The work of stabilizing the tests is expected to be the bulk of the
required time and effort.
On the 5.4 branch, bug fixes, test fixes, and documentation can go in
after review. New features and behavior changes need the consent of
the release manager.
PHASE 3 WORKSTREAMS
1. Test health
- Fix flaky tests, or quarantine them with a JIRA.
- Get the nightly results clean.
2. Feature hardening
- V2 WHERE optimizer: Add a CI run of the full integration test
suite with the V2 WHERE optimizer enabled.
- Consistent Failover: Run the cross-cluster integration tests.
Test a rolling upgrade while clients that use the ZooKeeper-
based HA connection continue to have service. Finish the HA client
documentation.
- Vector indexes: Make sure that each shaded jar preserves the multi-
release manifest.
3. Compatibility and upgrade
- Use the backward compatibility test framework (PHOENIX-7702).
Test 5.3 clients against 5.4 servers, and 5.4 clients against 5.3
servers.
- Test the upgrade of the SYSTEM tables from 5.3.
- Test a rolling upgrade of the region servers from 5.3 to 5.4.
4. Supported JDK matrix
- Build and test each HBase profile (2.5.0, 2.5, 2.6.0, 2.6) on JDKs
11, 17, and 21.
- Record the supported combinations in the release notes.
5. Dependencies and legal requirements
- Review LICENSE and NOTICE for the new dependencies and their
transitive dependencies.
- Do a CVE scan of the binary artifacts.
- Ensure Apache RAT audits are successful.
6. Release tooling
- The release image `dev/create-release/phoenix-rm/Dockerfile` uses
JDK 8. Move it to JDK 11 at least, although JDK 17 is recommended.
- Do a dry run of the release scripts before RC0.
7. Downstream projects
- Build `phoenix-queryserver` and `phoenix-connectors` against
`5.4.0-SNAPSHOT` with Java 11 or later and document any changes
required. Note that the 5.4.0 core release is not blocked by these
downstreamers, who can continue with 5.3.
8. Documentation
- The release notes will include these items:
- The new minimum Java version policy.
- The EXPLAIN format changes.
- The new features, their flags or configuration options, and
defaults.
- The supported HBase versions and JDKs.
- Update the website documentation for vector indexes, Consistent
Failover, and EXPLAIN improvements.
The exit criteria for Phase 3 are as follows:
- No release blockers are open.
- The nightly builds pass on the full platform matrix.
- The compatibility and upgrade tests pass.
- The release notes and documentation are complete.
- The release tooling dry run passes.
QUESTIONS
Do we agree on the scope?
Do we accept the minimum Java version policy for the 5.4 code line:
Java 11 minimum, Java 17 recommended, Java 21 suggested for vector
indexes?
Are there other current blockers in addition to PHOENIX-7279 and
PHOENIX-7596?
--
Best regards,
Andrew
