lhotari opened a new pull request, #4886: URL: https://github.com/apache/bookkeeper/pull/4886
### Motivation Upgrade RocksDB JNI from 9.9.3 to **10.10.1**, the latest version published on [Maven Central](https://repo.maven.apache.org/maven2/org/rocksdb/rocksdbjni/maven-metadata.xml) as checked on 2026-09-10, while preserving the bookie's RocksDB index data compatibility with **7.9.2**. BookKeeper [4.17.0](https://github.com/apache/bookkeeper/blob/release-4.17.0/pom.xml) uses 7.9.2; later 4.17.x patches use 7.10.2, so 7.9.2 is the conservative compatibility baseline for Pulsar 4.0.x deployments. Upstream's latest release is [11.8.1](https://github.com/facebook/rocksdb/releases/tag/v11.8.1), but its rocksdbjni artifact is not available on Maven Central. This PR deliberately targets the latest consumable JNI release, rather than introducing an unresolvable dependency. ### Changes - Upgrade `rocksdb.version` to 10.10.1. - Retain `format_version=5` in all three `conf/*_rocksdb.conf.default` templates and document the benefits and minimum reader versions of formats 6 and 7. - Explicitly set `track_and_verify_wals=false` to preserve WAL recovery by 7.9.2, with comments explaining when the newer integrity checks can be enabled. - Document compatible HyperClockCache, Ribbon filter, and partitioned-filter optimizations with upstream documentation links. Keep the existing checksum, compression and entry-location tuning. - Update the corresponding test configurations and add a test that writes/compacts data and checks the actual SST footer version for every database type, with both configuration files and Java defaults. ### Backwards compatibility at the storage format level **The default bookie ledger-metadata and entry-location indexes remain readable and writable by RocksDB 7.9.2 after upgrading to 10.10.1. No index migration or rewrite is required for this configuration.** This concerns RocksDB indexes; the BookKeeper journal and entry-log formats are unchanged. | Component | Compatibility decision | | --- | --- | | SST files | Continue writing format 5, readable since RocksDB 6.6. Format 6 requires >= 8.6 and adds location-dependent checksums/footer protection; format 7 requires >= 10.4 and supports custom compression. Neither is suitable while 7.9.2 rollback is required. See [table options](https://github.com/facebook/rocksdb/blob/v10.10.1/include/rocksdb/table.h). | | WAL | Keep `track_and_verify_wals=false` (also the native default used by the Java fallback). The feature introduced in 9.11 writes predecessor-WAL records that the [7.9.2 reader](https://github.com/facebook/rocksdb/blob/v7.9.2/db/log_reader.cc) does not recognize. Existing ordinary write batches remain recoverable. | | MANIFEST | Ordinary index metadata remains compatible. The 10.9 release fixes range-deletion boundary metadata that could previously persist a version-dependent internal value type and break older readers. Round-trip validation includes range deletion followed by compaction. | | Compression, checksums and filters | Existing LZ4/Snappy compression and xxHash checksums are readable by 7.9.2. Modern Bloom filters and optional Ribbon filters are supported. Decoupled partitioned filters, enabled by default since 10.6, retain the old readable format; disabling them would unnecessarily forgo optimizations. | Storage compatibility does **not** mean configuration files can be copied unchanged between releases: 7.9.2 rejects the new `track_and_verify_wals` option even when false. Use the older release's configuration on rollback, retaining compatible format/compression settings; do not load the newer generated `OPTIONS-*` file into the older strict parser. Operators using custom options must preserve these constraints. Lowering `format_version` does not convert SSTs already written in a newer format; those need rewriting with a capable RocksDB version before rollback. Enabling newer WAL records likewise cannot be undone merely by changing the option back. This compatibility statement is scoped to the bookie's `KeyValueStorageRocksDB` indexes and their supplied/default settings, not the separate stream state-store implementation or arbitrary opt-in RocksDB features. ### Release-note review Reviewed the [published release notes since 9.9.3](https://github.com/facebook/rocksdb/releases) and [cumulative history through 11.8.1](https://github.com/facebook/rocksdb/blob/v11.8.1/HISTORY.md), including patch-release notes. Relevant changes: - **9.10–9.11:** WAL tracking is opt-in; manual leveled compactions respect `max_compaction_bytes` more strictly. - **10.0–10.2:** removed obsolete compression/API options are not used by the supplied configuration; writing SST formats below 2 is unsupported; OPTIONS persistence errors now always fail the operation. Custom configurations must remove deleted options such as `fail_if_options_file_error` and `max_write_buffer_number_to_maintain`. - **10.4–10.6:** format 7 is opt-in; file-size validation on open is strengthened; periodic compaction scheduling changes; decoupled partitioned filters become the default without breaking SST compatibility. - **10.7–10.10:** native default cache changes to HyperClockCache, parallel compression improves, the LZ4 performance regression is fixed, MANIFEST auto-tuning is added, and the range-deletion compatibility bug is fixed. Explicit entry-location LRU caches remain explicit. These cache/compaction changes do not require an index-format migration. - **Beyond this PR (10.11–11.8.1):** format 7 becomes the upstream default; 11.0 drops reads of SST formats 0/1, which would require pre-upgrade compaction for such legacy files; 11.5 changes default compression to LZ4, still readable by 7.9.2. New custom compression, embedded blobs and other experimental formats are not enabled by this PR. A future 11.x JNI upgrade needs its own validation. ### Validation - Passed 13 focused tests, including all six database-type/configuration combinations in the new SST-format test: `mvn -B -pl bookkeeper-server,stream/statelib -am test -Dtest=KeyValueStorageRocksDBTest,KeyValueStorageTest,EntryLocationIndexTest,LedgerMetadataIndexTest -Dsurefire.failIfNoSpecifiedTests=false` - Passed `mvn -B -pl bookkeeper-server -am checkstyle:check apache-rat:check`. - Passed an additional standalone JNI test for each of the three templates: **7.9.2 → 10.10.1 → 7.9.2 → 10.10.1**, using separate JVMs/native libraries. Wrote 2,000 initial records, upgraded and applied point/range deletions plus 2,000 new records, flushed/compacted, then halted the JVM after an additional synced but unflushed write. The older version successfully recovered that WAL, verified all expected values/deletions, and wrote/compacted more data; the upgraded version then reopened and verified again. Older JVMs used the previous configuration files. - Full cluster upgrade/downgrade and the complete repository test suite were not run. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
