[
https://issues.apache.org/jira/browse/HDDS-15937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18128686#comment-18128686
]
Siyao Meng commented on HDDS-15937:
-----------------------------------
Follow-up: I ran a one-time downgrade test to cover what this validation did
not exercise: OM snapshots and snapshot diff, datanode schema V3 container
replication, the offline repair/debug tools, and the native rocks-tools
library. It found a downgrade bug that this test missed: HDDS-17151.
h3. Setup
* Cluster: {{upgrade/compose/ha}} (3 OM, 3 SCM, 5 datanodes, Recon, secure).
* Sequence: 2.2.1 (RocksDB 7.7.3) -> master + HDDS-17143 (RocksDB 11.8.1),
pre-finalized -> 2.2.1 -> 11.8.1 + finalize.
* The common upgrade callbacks (generate/validate) passed in every phase.
h3. Results
* *Snapshots: OK, except db.snapdiff (HDDS-17151).*
** 2.2.1 lists and reads snapshots created by 11.8.1.
** 2.2.1 returns the same snap1->snap2 diff report: +2001 -11 across 3 pages.
** 2.2.1 created snap3 and diffed snap2->snap3 and snap1->snap3.
** After re-upgrading, 11.8.1 read snap3 and diffed snap3->snap4 (+301) and
snap1->snap4 (+2292 -11, as expected). It iterated the delta SSTs with the
native library.
* *db.snapdiff: broken (HDDS-17151).*
** The OM that ran the snapshot diff on 11.8.1 (om3) failed to start on 2.2.1
with {{Corrupt or unsupported format_version: 7 in file
.../db.snapdiff/MANIFEST}}.
** {{OmSnapshotManager}} opens {{db.snapdiff}} without the {{format_version=5}}
pin from HDDS-14225.
** Master on 10.10.1.1 is affected the same way, with format_version 6.
* *Compaction:* {{ozone repair om compact}} passed on all OMs with 11.8.1, and
on the running OMs with 2.2.1.
* *Datanode schema V3 replication: OK.*
** 11.8.1 replicated a closed RATIS/ONE container away from a decommissioning
datanode. The key read back correctly with the source datanode stopped.
** After the downgrade, 2.2.1 read that replica, replicated it again, and
served the key with both earlier holders stopped. The md5 matched each time.
* *SST formats:* every SST written by 11.8.1 for OM, SCM and the datanode
container DB is format_version 5 with LZ4. The only exception is db.snapdiff.
* *Offline tools against DB copies last written by 11.8.1:*
** 2.2.1 {{ozone debug ldb scan}} works on the OM, snapshot and datanode DBs.
** 2.2.1 {{ozone repair ldb compact}} and {{ozone repair om
update-transaction}} fail with {{Could not find option: :
use_direct_io_for_compaction_reads}}. That is HDDS-17144.
** With the HDDS-17144 fix, master code (RocksDB 10.10.1.1) opens the same OM
DB copy via {{openWithLatestOptions}}.
** The same problem already exists between master (10.10.1.1) and 2.2.x
(7.7.3): {{Could not find option: : metadata_write_temperature}}.
h3. Side findings (not RocksDB format)
* *2.2.1 native library never loads.* The official 2.2.1 image cannot load its
own {{libozone_rocksdb_tools.so}}, so 2.2.1 always falls back to a full
snapshot diff.
** The library was built with Debian GCC 14.2 and needs {{GLIBCXX_3.4.32}} for
{{std::ios_base_library_init()}}, emitted because of an unused {{#include
<iostream>}}.
** Rocky 9 provides GLIBCXX_3.4.29.
* *Recon cannot be downgraded.* Recon 2.2.1 does not start after Recon 2.3.0
has run: {{Metadata layout version (101) > software layout version (10)}}.
> Verify RocksDB 7.7.3 in 2.2.0 can reopen the database after RocksDB 10.10.1.1
> wrote and compacted SSTs
> ------------------------------------------------------------------------------------------------------
>
> Key: HDDS-15937
> URL: https://issues.apache.org/jira/browse/HDDS-15937
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: test
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Attachments: HDDS-15937-rocksdb-rollback-validation.patch
>
>
> h3. Goal
> Verify that Ozone 2.2.0 with RocksDB 7.7.3 can reopen and read the same OM
> database after Ozone 2.3.0 with RocksDB 10.10.1.1 has written and compacted
> new SST files. The new SSTs must remain at block-based table format_version 5
> so rollback to 2.2.0 remains compatible.
> h3. Test command
> {code:java}
> run_test ha non-rolling-upgrade 2.2.0 2.3.0
> {code}
> h3. Test scenario
> # Start the HA cluster on Ozone 2.2.0, verify it is finalized, and create
> the baseline upgrade dataset.
> # Restart the same cluster and data directories on Ozone 2.3.0 with
> --upgrade. Do not invoke the explicit finalization step before downgrade.
> Since 2.3.0 introduces no new OM or HDDS layout feature after 2.2.0, OM and
> SCM already report finalized; this exercises rollback across the binary and
> RocksDB version change rather than rollback with a pending layout
> finalization.
> # While running 2.3.0, create a dedicated OBS volume and bucket and perform
> three rounds of writes. Each round writes 2,000 keys and forces keyTable
> compaction on om1, om2, and om3. Inventory newly created SST files, copy them
> to the test results, inspect their table properties, and require
> format_version 5. Require at least one newly created OM SST. Newly observed
> SCM and datanode SSTs are checked opportunistically.
> # Stop 2.3.0 and restart the same data directories on Ozone 2.2.0 with
> --downgrade. Validate the common datasets written under 2.3.0 and read
> format-round-3/0, written during the final 2.3.0 compaction round. Require
> the downloaded key to be non-empty.
> # Restart 2.3.0, run the normal explicit finalization phase, and complete
> the common upgrade validations.
> h3. Success criteria
> * Compaction completes successfully on all three OMs in all three rounds.
> * Every newly observed SST reports format_version 5.
> * At least one new OM SST is observed.
> * Ozone 2.2.0 with RocksDB 7.7.3 starts successfully against the database
> previously used by 2.3.0.
> * The downgraded 2.2.0 cluster reads both the normal upgrade datasets and
> format-round-3/0.
> * The complete HA non-rolling-upgrade test passes.
> h3. Evidence from the ad-hoc validation
> * {color:#FF0000}+Nine new OM SST files were observed across three
> compaction rounds; all reported format_version 5.+{color}
> * {color:#FF0000}+After downgrade, Ozone 2.2.0 successfully read
> format-round-3/0.+{color}
> * The complete Robot Framework run finished with 77 passed, 0 failed.
> h3. Proof boundary
> This is an end-to-end database reopen and data-read test: RocksDB 7.7.3
> starts against the same database after RocksDB 10.10.1.1 wrote and compacted
> SSTs, then serves data written by 2.3.0. The format inspector reads each
> copied SST with RocksDB 10.10.1.1 to verify format_version 5. The test does
> not separately open every copied SST with RocksDB 7.7.3, and it does not map
> format-round-3/0 to one specific inspected SST.
> h3. Reproducing the validation
> The attached [^HDDS-15937-rocksdb-rollback-validation.patch] is the output of
> git diff --cached --binary for the ad-hoc validation changes. It is based on
> HDDS-15769 ([PR #10836|https://github.com/apache/ozone/pull/10836]), which
> adds the preceding upgrade callbacks and test-matrix context. Apply PR #10836
> first; the attached patch is not intended to apply directly to asf/master
> without that prerequisite.
> Attachment SHA-256:
> {noformat}
> 8743eed9d4f51295532a153230c79f6a03203287767af18a4c44c8d17801d5cf{noformat}
> h4. Apply the prerequisite and validation patch
> Start from a clean checkout of the desired asf/master revision. Download the
> attached patch from this Jira, then run:
> {code:bash}
> git status --short
> curl -L https://github.com/apache/ozone/pull/10836.diff | git apply -
> git apply --check /path/to/HDDS-15937-rocksdb-rollback-validation.patch
> git apply /path/to/HDDS-15937-rocksdb-rollback-validation.patch
> {code}
> h4. Build and run
> Docker with Compose must be available. Use JDK 21. The exact ad-hoc
> validation used a locally built
> {code:java}
> apache/ozone:2.3.0-rocky{code}
> binary image containing the latest
> {code:java}
> asf/master{code}
> binaries. The test invocation below deliberately resets
> {code:java}
> OZONE_CURRENT_VERSION=src{code}
> after loading the generated test library; this makes the explicit target
> {code:java}
> 2.3.0{code}
> select the binary image instead of the local
> {code:java}
> apache/ozone-runner{code}
> path.
> Both
> {code:java}
> apache/ozone:2.2.0-rocky{code}
> and
> {code:java}
> apache/ozone:2.3.0-rocky{code}
> must exist locally before running the exact binary-image test. The following
> builds the 2.3.0 image from the same latest-master checkout to which PR
> #10836 and the attached validation patch were applied. Prepare the 2.2.0
> image separately if it is not already present.
> {code:bash}
> export JAVA_HOME=/path/to/jdk-21
> export PATH="$JAVA_HOME/bin:$PATH"
> mvn -q dependency:get -Dartifact=org.rocksdb:rocksdbjni:10.10.1.1
> mvn -Pdist -DskipTests package
> MASTER_REV=$(git rev-parse HEAD)
> DOCKER_CONTEXT=$(mktemp -d)
> git clone https://github.com/apache/ozone-docker.git "$DOCKER_CONTEXT"
> git -C "$DOCKER_CONTEXT" checkout 6d91ca5a3d9a8ff68227854938526f3a6cbccc6c
> cp hadoop-ozone/dist/target/ozone-2.3.0-SNAPSHOT.tar.gz "$DOCKER_CONTEXT"/
> git -C "$DOCKER_CONTEXT" apply <<'PATCH'
> diff --git a/Dockerfile b/Dockerfile
> --- a/Dockerfile
> +++ b/Dockerfile
> @@ -16,10 +16,8 @@
> ARG OZONE_RUNNER_VERSION=20260626-1-jdk21
> FROM ${OZONE_RUNNER_IMAGE}:${OZONE_RUNNER_VERSION}
>
> -ARG OZONE_VERSION=2.1.1
> -ARG
> OZONE_URL="https://www.apache.org/dyn/closer.lua?action=download&filename=ozone/${OZONE_VERSION}/ozone-${OZONE_VERSION}.tar.gz";
> -
> WORKDIR /opt
> -RUN sudo rm -rf /opt/hadoop && curl -LSs -o ozone.tar.gz $OZONE_URL && tar
> zxf ozone.tar.gz && rm ozone.tar.gz && mv ozone* hadoop
> +COPY ozone-2.3.0-SNAPSHOT.tar.gz ozone.tar.gz
> +RUN sudo rm -rf /opt/hadoop && tar zxf ozone.tar.gz && rm ozone.tar.gz && mv
> ozone-* hadoop
>
> WORKDIR /opt/hadoop
> PATCH
> docker build \
> --build-arg OZONE_RUNNER_VERSION=20260626-1-jdk25 \
> --label "org.opencontainers.image.revision=$MASTER_REV" \
> -t apache/ozone:2.3.0-rocky \
> "$DOCKER_CONTEXT"
> docker image inspect apache/ozone:2.3.0-rocky \
> --format '{{index .Config.Labels "org.opencontainers.image.revision"}}'
> docker run --rm apache/ozone:2.3.0-rocky ozone version
> docker image inspect apache/ozone:2.2.0-rocky >/dev/null
> {code}
> Run only the requested binary-image upgrade test:
> {code:bash}
> export
> ROCKSDBJNI_JAR="$HOME/.m2/repository/org/rocksdb/rocksdbjni/10.10.1.1/rocksdbjni-10.10.1.1.jar"
> test -f "$ROCKSDBJNI_JAR"
> docker info >/dev/null
> cd hadoop-ozone/dist/target/ozone-2.3.0-SNAPSHOT/compose/upgrade
> bash -c '
> export TEST_DIR="$PWD" COMPOSE_DIR="$PWD"
> source ./testlib.sh
> export OZONE_CURRENT_VERSION=src
> RESULT_DIR="$ALL_RESULT_DIR" create_results_dir
> run_test ha non-rolling-upgrade 2.2.0 2.3.0
> generate_report upgrade "$ALL_RESULT_DIR"
> exit "$RESULT"
> '
> {code}
> The simpler
> {code:java}
> ./test.sh{code}
> path uses the locally built 2.3.0 distribution through
> {code:java}
> apache/ozone-runner{code}
> ; it does not consume
> {code:java}
> apache/ozone:2.3.0-rocky{code}
> . The explicit invocation above matches the original validation and proves
> that the locally built 2.3.0 binary image is the image under test.
> Inspect
> {code:java}
> result/ha-non-rolling-upgrade-2.2.0-2.3.0/report.html{code}
> and
> {code:java}
> result/ha-non-rolling-upgrade-2.2.0-2.3.0/sst-format/formats.tsv{code}
> . A successful run completes all four upgrade phases, reports
> {code:java}
> format_version 5{code}
> for every newly observed SST, observes at least one new OM SST, and reads
> {code:java}
> format-round-3/0{code}
> after downgrade to 2.2.0.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
