[
https://issues.apache.org/jira/browse/HDDS-15937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-15937:
------------------------------
Description:
h3. Goal
Verify that Ozone 2.2.0 with RocksDB 7.7.3 can reopen and read the same OM
database after Ozone 2.3.0 with RocksDB 10.10.1.1 has written and compacted new
SST files. The new SSTs must remain at block-based table format_version 5 so
rollback to 2.2.0 remains compatible.
h3. Test command
{code}
run_test ha non-rolling-upgrade 2.2.0 2.3.0
{code}
h3. Test scenario
# Start the HA cluster on Ozone 2.2.0, verify it is finalized, and create the
baseline upgrade dataset.
# Restart the same cluster and data directories on Ozone 2.3.0 with --upgrade.
Do not invoke the explicit finalization step before downgrade. Since 2.3.0
introduces no new OM or HDDS layout feature after 2.2.0, OM and SCM already
report finalized; this exercises rollback across the binary and RocksDB version
change rather than rollback with a pending layout finalization.
# While running 2.3.0, create a dedicated OBS volume and bucket and perform
three rounds of writes. Each round writes 2,000 keys and forces keyTable
compaction on om1, om2, and om3. Inventory newly created SST files, copy them
to the test results, inspect their table properties, and require format_version
5. Require at least one newly created OM SST. Newly observed SCM and datanode
SSTs are checked opportunistically.
# Stop 2.3.0 and restart the same data directories on Ozone 2.2.0 with
--downgrade. Validate the common datasets written under 2.3.0 and read
format-round-3/0, written during the final 2.3.0 compaction round. Require the
downloaded key to be non-empty.
# Restart 2.3.0, run the normal explicit finalization phase, and complete the
common upgrade validations.
h3. Success criteria
* Compaction completes successfully on all three OMs in all three rounds.
* Every newly observed SST reports format_version 5.
* At least one new OM SST is observed.
* Ozone 2.2.0 with RocksDB 7.7.3 starts successfully against the database
previously used by 2.3.0.
* The downgraded 2.2.0 cluster reads both the normal upgrade datasets and
format-round-3/0.
* The complete HA non-rolling-upgrade test passes.
h3. Evidence from the ad-hoc validation
* Nine new OM SST files were observed across three compaction rounds; all
reported format_version 5.
* After downgrade, Ozone 2.2.0 successfully read format-round-3/0.
* The complete Robot Framework run finished with 77 passed, 0 failed.
h3. Proof boundary
This is an end-to-end database reopen and data-read test: RocksDB 7.7.3 starts
against the same database after RocksDB 10.10.1.1 wrote and compacted SSTs,
then serves data written by 2.3.0. The format inspector reads each copied SST
with RocksDB 10.10.1.1 to verify format_version 5. The test does not separately
open every copied SST with RocksDB 7.7.3, and it does not map format-round-3/0
to one specific inspected SST.
h3. Reproducing the validation
The attached [^HDDS-15937-rocksdb-rollback-validation.patch] is the output of
git diff --cached --binary for the ad-hoc validation changes. It is based on
HDDS-15769 ([PR #10836|https://github.com/apache/ozone/pull/10836]), which adds
the preceding upgrade callbacks and test-matrix context. Apply PR #10836 first;
the attached patch is not intended to apply directly to asf/master without that
prerequisite.
Attachment SHA-256:
{noformat}8743eed9d4f51295532a153230c79f6a03203287767af18a4c44c8d17801d5cf{noformat}
h4. Apply the prerequisite and validation patch
Start from a clean checkout of the desired asf/master revision. Download the
attached patch from this Jira, then run:
{code:bash}
git status --short
curl -L https://github.com/apache/ozone/pull/10836.diff | git apply -
git apply --check /path/to/HDDS-15937-rocksdb-rollback-validation.patch
git apply /path/to/HDDS-15937-rocksdb-rollback-validation.patch
{code}
h4. Build and run
Docker with Compose must be available. Use JDK 21 and build the 2.3.0-SNAPSHOT
distribution. The upgrade harness uses that local distribution through the
ozone-runner image and uses the released apache/ozone:2.2.0-rocky image for the
old version.
{code:bash}
export JAVA_HOME=/path/to/jdk-21
export PATH="$JAVA_HOME/bin:$PATH"
mvn -q dependency:get -Dartifact=org.rocksdb:rocksdbjni:10.10.1.1
mvn -Pdist -DskipTests package
export
ROCKSDBJNI_JAR="$HOME/.m2/repository/org/rocksdb/rocksdbjni/10.10.1.1/rocksdbjni-10.10.1.1.jar"
test -f "$ROCKSDBJNI_JAR"
docker info >/dev/null
cd hadoop-ozone/dist/target/ozone-2.3.0-SNAPSHOT/compose/upgrade
./test.sh
{code}
The active test.sh entry runs the following upgrade against the locally built
2.3.0 version:
{code}
run_test ha non-rolling-upgrade 2.2.0 2.3.0
{code}
Inspect result/ha-non-rolling-upgrade-2.2.0-2.3.0/report.html and
result/ha-non-rolling-upgrade-2.2.0-2.3.0/sst-format/formats.tsv. A successful
run completes all four upgrade phases, reports format_version 5 for every newly
observed SST, observes at least one new OM SST, and reads format-round-3/0
after downgrade to 2.2.0.
was:
h3. Goal
Verify that Ozone 2.2.0 with RocksDB 7.7.3 can reopen and read the same OM
database after Ozone 2.3.0 with RocksDB 10.10.1.1 has written and compacted new
SST files. The new SSTs must remain at block-based table format_version 5 so
rollback to 2.2.0 remains compatible.
h3. Test command
{code}
run_test ha non-rolling-upgrade 2.2.0 2.3.0
{code}
h3. Test scenario
# Start the HA cluster on Ozone 2.2.0, verify it is finalized, and create the
baseline upgrade dataset.
# Restart the same cluster and data directories on Ozone 2.3.0 with --upgrade.
Do not invoke the explicit finalization step before downgrade. Since 2.3.0
introduces no new OM or HDDS layout feature after 2.2.0, OM and SCM already
report finalized; this exercises rollback across the binary and RocksDB version
change rather than rollback with a pending layout finalization.
# While running 2.3.0, create a dedicated OBS volume and bucket and perform
three rounds of writes. Each round writes 2,000 keys and forces keyTable
compaction on om1, om2, and om3. Inventory newly created SST files, copy them
to the test results, inspect their table properties, and require format_version
5. Require at least one newly created OM SST. Newly observed SCM and datanode
SSTs are checked opportunistically.
# Stop 2.3.0 and restart the same data directories on Ozone 2.2.0 with
--downgrade. Validate the common datasets written under 2.3.0 and read
format-round-3/0, written during the final 2.3.0 compaction round. Require the
downloaded key to be non-empty.
# Restart 2.3.0, run the normal explicit finalization phase, and complete the
common upgrade validations.
h3. Success criteria
* Compaction completes successfully on all three OMs in all three rounds.
* Every newly observed SST reports format_version 5.
* At least one new OM SST is observed.
* Ozone 2.2.0 with RocksDB 7.7.3 starts successfully against the database
previously used by 2.3.0.
* The downgraded 2.2.0 cluster reads both the normal upgrade datasets and
format-round-3/0.
* The complete HA non-rolling-upgrade test passes.
h3. Evidence from the ad-hoc validation
* Nine new OM SST files were observed across three compaction rounds; all
reported format_version 5.
* After downgrade, Ozone 2.2.0 successfully read format-round-3/0.
* The complete Robot Framework run finished with 77 passed, 0 failed.
h3. Proof boundary
This is an end-to-end database reopen and data-read test: RocksDB 7.7.3 starts
against the same database after RocksDB 10.10.1.1 wrote and compacted SSTs,
then serves data written by 2.3.0. The format inspector reads each copied SST
with RocksDB 10.10.1.1 to verify format_version 5. The test does not separately
open every copied SST with RocksDB 7.7.3, and it does not map format-round-3/0
to one specific inspected SST.
> Verify RocksDB 7.7.3 in 2.2.0 can reopen the database after RocksDB 10.10.1.1
> wrote and compacted SSTs
> ------------------------------------------------------------------------------------------------------
>
> Key: HDDS-15937
> URL: https://issues.apache.org/jira/browse/HDDS-15937
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: test
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Attachments: HDDS-15937-rocksdb-rollback-validation.patch
>
>
> h3. Goal
> Verify that Ozone 2.2.0 with RocksDB 7.7.3 can reopen and read the same OM
> database after Ozone 2.3.0 with RocksDB 10.10.1.1 has written and compacted
> new SST files. The new SSTs must remain at block-based table format_version 5
> so rollback to 2.2.0 remains compatible.
> h3. Test command
> {code}
> run_test ha non-rolling-upgrade 2.2.0 2.3.0
> {code}
> h3. Test scenario
> # Start the HA cluster on Ozone 2.2.0, verify it is finalized, and create the
> baseline upgrade dataset.
> # Restart the same cluster and data directories on Ozone 2.3.0 with
> --upgrade. Do not invoke the explicit finalization step before downgrade.
> Since 2.3.0 introduces no new OM or HDDS layout feature after 2.2.0, OM and
> SCM already report finalized; this exercises rollback across the binary and
> RocksDB version change rather than rollback with a pending layout
> finalization.
> # While running 2.3.0, create a dedicated OBS volume and bucket and perform
> three rounds of writes. Each round writes 2,000 keys and forces keyTable
> compaction on om1, om2, and om3. Inventory newly created SST files, copy them
> to the test results, inspect their table properties, and require
> format_version 5. Require at least one newly created OM SST. Newly observed
> SCM and datanode SSTs are checked opportunistically.
> # Stop 2.3.0 and restart the same data directories on Ozone 2.2.0 with
> --downgrade. Validate the common datasets written under 2.3.0 and read
> format-round-3/0, written during the final 2.3.0 compaction round. Require
> the downloaded key to be non-empty.
> # Restart 2.3.0, run the normal explicit finalization phase, and complete the
> common upgrade validations.
> h3. Success criteria
> * Compaction completes successfully on all three OMs in all three rounds.
> * Every newly observed SST reports format_version 5.
> * At least one new OM SST is observed.
> * Ozone 2.2.0 with RocksDB 7.7.3 starts successfully against the database
> previously used by 2.3.0.
> * The downgraded 2.2.0 cluster reads both the normal upgrade datasets and
> format-round-3/0.
> * The complete HA non-rolling-upgrade test passes.
> h3. Evidence from the ad-hoc validation
> * Nine new OM SST files were observed across three compaction rounds; all
> reported format_version 5.
> * After downgrade, Ozone 2.2.0 successfully read format-round-3/0.
> * The complete Robot Framework run finished with 77 passed, 0 failed.
> h3. Proof boundary
> This is an end-to-end database reopen and data-read test: RocksDB 7.7.3
> starts against the same database after RocksDB 10.10.1.1 wrote and compacted
> SSTs, then serves data written by 2.3.0. The format inspector reads each
> copied SST with RocksDB 10.10.1.1 to verify format_version 5. The test does
> not separately open every copied SST with RocksDB 7.7.3, and it does not map
> format-round-3/0 to one specific inspected SST.
> h3. Reproducing the validation
> The attached [^HDDS-15937-rocksdb-rollback-validation.patch] is the output of
> git diff --cached --binary for the ad-hoc validation changes. It is based on
> HDDS-15769 ([PR #10836|https://github.com/apache/ozone/pull/10836]), which
> adds the preceding upgrade callbacks and test-matrix context. Apply PR #10836
> first; the attached patch is not intended to apply directly to asf/master
> without that prerequisite.
> Attachment SHA-256:
> {noformat}8743eed9d4f51295532a153230c79f6a03203287767af18a4c44c8d17801d5cf{noformat}
> h4. Apply the prerequisite and validation patch
> Start from a clean checkout of the desired asf/master revision. Download the
> attached patch from this Jira, then run:
> {code:bash}
> git status --short
> curl -L https://github.com/apache/ozone/pull/10836.diff | git apply -
> git apply --check /path/to/HDDS-15937-rocksdb-rollback-validation.patch
> git apply /path/to/HDDS-15937-rocksdb-rollback-validation.patch
> {code}
> h4. Build and run
> Docker with Compose must be available. Use JDK 21 and build the
> 2.3.0-SNAPSHOT distribution. The upgrade harness uses that local distribution
> through the ozone-runner image and uses the released apache/ozone:2.2.0-rocky
> image for the old version.
> {code:bash}
> export JAVA_HOME=/path/to/jdk-21
> export PATH="$JAVA_HOME/bin:$PATH"
> mvn -q dependency:get -Dartifact=org.rocksdb:rocksdbjni:10.10.1.1
> mvn -Pdist -DskipTests package
> export
> ROCKSDBJNI_JAR="$HOME/.m2/repository/org/rocksdb/rocksdbjni/10.10.1.1/rocksdbjni-10.10.1.1.jar"
> test -f "$ROCKSDBJNI_JAR"
> docker info >/dev/null
> cd hadoop-ozone/dist/target/ozone-2.3.0-SNAPSHOT/compose/upgrade
> ./test.sh
> {code}
> The active test.sh entry runs the following upgrade against the locally built
> 2.3.0 version:
> {code}
> run_test ha non-rolling-upgrade 2.2.0 2.3.0
> {code}
> Inspect result/ha-non-rolling-upgrade-2.2.0-2.3.0/report.html and
> result/ha-non-rolling-upgrade-2.2.0-2.3.0/sst-format/formats.tsv. A
> successful run completes all four upgrade phases, reports format_version 5
> for every newly observed SST, observes at least one new OM SST, and reads
> format-round-3/0 after downgrade to 2.2.0.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
