[
https://issues.apache.org/jira/browse/HDDS-15937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng resolved HDDS-15937.
-------------------------------
Resolution: Done
> Verify RocksDB 7.7.3 in 2.2.0 can reopen the database after RocksDB 10.10.1.1
> wrote and compacted SSTs
> ------------------------------------------------------------------------------------------------------
>
> Key: HDDS-15937
> URL: https://issues.apache.org/jira/browse/HDDS-15937
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: test
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Attachments: HDDS-15937-rocksdb-rollback-validation.patch
>
>
> h3. Goal
> Verify that Ozone 2.2.0 with RocksDB 7.7.3 can reopen and read the same OM
> database after Ozone 2.3.0 with RocksDB 10.10.1.1 has written and compacted
> new SST files. The new SSTs must remain at block-based table format_version 5
> so rollback to 2.2.0 remains compatible.
> h3. Test command
> {code}
> run_test ha non-rolling-upgrade 2.2.0 2.3.0
> {code}
> h3. Test scenario
> # Start the HA cluster on Ozone 2.2.0, verify it is finalized, and create the
> baseline upgrade dataset.
> # Restart the same cluster and data directories on Ozone 2.3.0 with
> --upgrade. Do not invoke the explicit finalization step before downgrade.
> Since 2.3.0 introduces no new OM or HDDS layout feature after 2.2.0, OM and
> SCM already report finalized; this exercises rollback across the binary and
> RocksDB version change rather than rollback with a pending layout
> finalization.
> # While running 2.3.0, create a dedicated OBS volume and bucket and perform
> three rounds of writes. Each round writes 2,000 keys and forces keyTable
> compaction on om1, om2, and om3. Inventory newly created SST files, copy them
> to the test results, inspect their table properties, and require
> format_version 5. Require at least one newly created OM SST. Newly observed
> SCM and datanode SSTs are checked opportunistically.
> # Stop 2.3.0 and restart the same data directories on Ozone 2.2.0 with
> --downgrade. Validate the common datasets written under 2.3.0 and read
> format-round-3/0, written during the final 2.3.0 compaction round. Require
> the downloaded key to be non-empty.
> # Restart 2.3.0, run the normal explicit finalization phase, and complete the
> common upgrade validations.
> h3. Success criteria
> * Compaction completes successfully on all three OMs in all three rounds.
> * Every newly observed SST reports format_version 5.
> * At least one new OM SST is observed.
> * Ozone 2.2.0 with RocksDB 7.7.3 starts successfully against the database
> previously used by 2.3.0.
> * The downgraded 2.2.0 cluster reads both the normal upgrade datasets and
> format-round-3/0.
> * The complete HA non-rolling-upgrade test passes.
> h3. Evidence from the ad-hoc validation
> * Nine new OM SST files were observed across three compaction rounds; all
> reported format_version 5.
> * After downgrade, Ozone 2.2.0 successfully read format-round-3/0.
> * The complete Robot Framework run finished with 77 passed, 0 failed.
> h3. Proof boundary
> This is an end-to-end database reopen and data-read test: RocksDB 7.7.3
> starts against the same database after RocksDB 10.10.1.1 wrote and compacted
> SSTs, then serves data written by 2.3.0. The format inspector reads each
> copied SST with RocksDB 10.10.1.1 to verify format_version 5. The test does
> not separately open every copied SST with RocksDB 7.7.3, and it does not map
> format-round-3/0 to one specific inspected SST.
> h3. Reproducing the validation
> The attached [^HDDS-15937-rocksdb-rollback-validation.patch] is the output of
> git diff --cached --binary for the ad-hoc validation changes. It is based on
> HDDS-15769 ([PR #10836|https://github.com/apache/ozone/pull/10836]), which
> adds the preceding upgrade callbacks and test-matrix context. Apply PR #10836
> first; the attached patch is not intended to apply directly to asf/master
> without that prerequisite.
> Attachment SHA-256:
> {noformat}8743eed9d4f51295532a153230c79f6a03203287767af18a4c44c8d17801d5cf{noformat}
> h4. Apply the prerequisite and validation patch
> Start from a clean checkout of the desired asf/master revision. Download the
> attached patch from this Jira, then run:
> {code:bash}
> git status --short
> curl -L https://github.com/apache/ozone/pull/10836.diff | git apply -
> git apply --check /path/to/HDDS-15937-rocksdb-rollback-validation.patch
> git apply /path/to/HDDS-15937-rocksdb-rollback-validation.patch
> {code}
> h4. Build and run
> Docker with Compose must be available. Use JDK 21. The exact ad-hoc
> validation used a locally built {code}apache/ozone:2.3.0-rocky{code} binary
> image containing the latest {code}asf/master{code} binaries. The test
> invocation below deliberately resets {code}OZONE_CURRENT_VERSION=src{code}
> after loading the generated test library; this makes the explicit target
> {code}2.3.0{code} select the binary image instead of the local
> {code}apache/ozone-runner{code} path.
> Both {code}apache/ozone:2.2.0-rocky{code} and
> {code}apache/ozone:2.3.0-rocky{code} must exist locally before running the
> exact binary-image test. The following builds the 2.3.0 image from the same
> latest-master checkout to which PR #10836 and the attached validation patch
> were applied. Prepare the 2.2.0 image separately if it is not already present.
> {code:bash}
> export JAVA_HOME=/path/to/jdk-21
> export PATH="$JAVA_HOME/bin:$PATH"
> mvn -q dependency:get -Dartifact=org.rocksdb:rocksdbjni:10.10.1.1
> mvn -Pdist -DskipTests package
> MASTER_REV=$(git rev-parse HEAD)
> DOCKER_CONTEXT=$(mktemp -d)
> git clone https://github.com/apache/ozone-docker.git "$DOCKER_CONTEXT"
> git -C "$DOCKER_CONTEXT" checkout 6d91ca5a3d9a8ff68227854938526f3a6cbccc6c
> cp hadoop-ozone/dist/target/ozone-2.3.0-SNAPSHOT.tar.gz "$DOCKER_CONTEXT"/
> git -C "$DOCKER_CONTEXT" apply <<'PATCH'
> diff --git a/Dockerfile b/Dockerfile
> --- a/Dockerfile
> +++ b/Dockerfile
> @@ -16,10 +16,8 @@
> ARG OZONE_RUNNER_VERSION=20260626-1-jdk21
> FROM ${OZONE_RUNNER_IMAGE}:${OZONE_RUNNER_VERSION}
>
> -ARG OZONE_VERSION=2.1.1
> -ARG
> OZONE_URL="https://www.apache.org/dyn/closer.lua?action=download&filename=ozone/${OZONE_VERSION}/ozone-${OZONE_VERSION}.tar.gz";
> -
> WORKDIR /opt
> -RUN sudo rm -rf /opt/hadoop && curl -LSs -o ozone.tar.gz $OZONE_URL && tar
> zxf ozone.tar.gz && rm ozone.tar.gz && mv ozone* hadoop
> +COPY ozone-2.3.0-SNAPSHOT.tar.gz ozone.tar.gz
> +RUN sudo rm -rf /opt/hadoop && tar zxf ozone.tar.gz && rm ozone.tar.gz && mv
> ozone-* hadoop
>
> WORKDIR /opt/hadoop
> PATCH
> docker build \
> --build-arg OZONE_RUNNER_VERSION=20260626-1-jdk25 \
> --label "org.opencontainers.image.revision=$MASTER_REV" \
> -t apache/ozone:2.3.0-rocky \
> "$DOCKER_CONTEXT"
> docker image inspect apache/ozone:2.3.0-rocky \
> --format '{{index .Config.Labels "org.opencontainers.image.revision"}}'
> docker run --rm apache/ozone:2.3.0-rocky ozone version
> docker image inspect apache/ozone:2.2.0-rocky >/dev/null
> {code}
> Run only the requested binary-image upgrade test:
> {code:bash}
> export
> ROCKSDBJNI_JAR="$HOME/.m2/repository/org/rocksdb/rocksdbjni/10.10.1.1/rocksdbjni-10.10.1.1.jar"
> test -f "$ROCKSDBJNI_JAR"
> docker info >/dev/null
> cd hadoop-ozone/dist/target/ozone-2.3.0-SNAPSHOT/compose/upgrade
> bash -c '
> export TEST_DIR="$PWD" COMPOSE_DIR="$PWD"
> source ./testlib.sh
> export OZONE_CURRENT_VERSION=src
> RESULT_DIR="$ALL_RESULT_DIR" create_results_dir
> run_test ha non-rolling-upgrade 2.2.0 2.3.0
> generate_report upgrade "$ALL_RESULT_DIR"
> exit "$RESULT"
> '
> {code}
> The simpler {code}./test.sh{code} path uses the locally built 2.3.0
> distribution through {code}apache/ozone-runner{code}; it does not consume
> {code}apache/ozone:2.3.0-rocky{code}. The explicit invocation above matches
> the original validation and proves that the locally built 2.3.0 binary image
> is the image under test.
> Inspect {code}result/ha-non-rolling-upgrade-2.2.0-2.3.0/report.html{code} and
> {code}result/ha-non-rolling-upgrade-2.2.0-2.3.0/sst-format/formats.tsv{code}.
> A successful run completes all four upgrade phases, reports
> {code}format_version 5{code} for every newly observed SST, observes at least
> one new OM SST, and reads {code}format-round-3/0{code} after downgrade to
> 2.2.0.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
