Siyao Meng created HDDS-17143:
---------------------------------
Summary: Upgrade RocksDB from 10.10.1.1 to 11.8.1
Key: HDDS-17143
URL: https://issues.apache.org/jira/browse/HDDS-17143
Project: Apache Ozone
Issue Type: Improvement
Reporter: Siyao Meng
Assignee: Siyao Meng
h3. Goal
Upgrade rocksdbjni from 10.10.1.1 (HDDS-14225) to 11.8.1, the latest rocksdbjni
release. Upstream published no jars for 11.2 to 11.7.
h3. Changes
* Bump {{rocksdb.version}} to 11.8.1.
* rocks-native: RocksDB 11.0 removed {{kLegacyBlockBasedTableMagicNumber}} when
it dropped {{format_version}} < 2. Mirror upstream {{sst_file_dumper.cc}} in
{{rocks-native.patch}}. Without this, the build still succeeds, but
{{libozone_rocksdb_tools.so}} fails to load with {{undefined symbol:
_ZN7rocksdb33kLegacyBlockBasedTableMagicNumberE}}.
* rocks-native: the rocksdbjni 11.x native library uses the C++11
{{std::string}} ABI. 10.10.1.1 used the old ABI. Stop forcing
{{-D_GLIBCXX_USE_CXX11_ABI=0}} in {{rocks-native/pom.xml}} and
{{CMakeLists.txt}}. Otherwise symbols such as
{{rocksdb::Comparator::CreateFromString}} do not resolve.
h3. Compatibility
* *Java API:* only {{DBOptions#setSkipCheckingSstFileSizesOnDbOpen}} and
{{ReadOptions#setManaged}} were removed. Ozone uses neither, and the whole
build, including test sources, compiles unchanged.
* *On-disk format:* {{format_version}} stays pinned to 5.
* *Downgrade:* verified that RocksDB 10.10.1.1 and 7.7.3 open, read,
{{verifyChecksum}} and compact a DB written by 11.8.1. The DB contained LZ4
SSTs, WAL-only data, deletes, a fixed-length prefix extractor CF, and 11.x
MANIFEST records. The new MANIFEST tags are in the safe-ignore ranges.
* *Default compression:* RocksDB 11.5 changed the default compression for
column families that do not set it from Snappy to LZ4 (facebook/rocksdb#14818).
New SSTs written by OM, SCM and datanode will be LZ4. Old and new versions read
both.
* *Platform:* the bundled {{librocksdbjni-linux64.so}} now needs
{{GLIBCXX_3.4.22}} (libstdc++ from GCC 6 or later). 10.10.1.1 needed
{{GLIBCXX_3.4.19}}. glibc stays at 2.17. Hosts with RHEL/CentOS 7 system
libstdc++ cannot load it.
* Do not use rocksdbjni 11.1.0/11.1.1. In those versions, closing a read-only
DB deletes live SST files of a concurrent read-write DB on the same directory
(introduced by facebook/rocksdb#14322, fixed by facebook/rocksdb#14881).
10.10.1.1, 11.1.2 and 11.8.1 are not affected (reproduced).
h3. Testing done locally
* Full build with test sources: OK.
* hdds-managed-rocksdb, rocksdb-checkpoint-differ and hdds-server-framework
unit tests: 249 run, 0 failures.
* rocks-native with {{-Drocks_tools_native}}: 46 run, 0 failures.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
