This is an automated email from the ASF dual-hosted git repository.
cadonna pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/kafka.git
The following commit(s) were added to refs/heads/trunk by this push:
new 8b72204bfdf KAFKA-15443: Upgrade RocksDB to 9.7.3 (#18275)
8b72204bfdf is described below
commit 8b72204bfdf1262ca8cb3c3a64a6c09aad13a5fc
Author: Swikar Patel <[email protected]>
AuthorDate: Thu Jan 9 02:55:56 2025 -0800
KAFKA-15443: Upgrade RocksDB to 9.7.3 (#18275)
This PR upgrades RocksDB from 7.9.2 to 9.7.3 and addresses the following
compatibility issues introduced by the RocksDB upgrade:
- Removal of AccessHint: The AccessHint class was completely removed in
RocksDB 9.7.3. This required removing all import statements, variable
declarations, method parameters, method return types, and static method calls
related to AccessHint in
RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter.java
RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.java Unused
methods are removed in
RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter.java
- Removal of NO_FILE_CLOSES: The NO_FILE_CLOSES metric was also removed in
RocksDB 9.7.3. The calculation for numberOfOpenFiles in
RocksDBMetricsRecorder.java has been adjusted to now track the total number of
file opens since the last reset. The previous calculation, which subtracted
NO_FILE_CLOSES from NO_FILE_OPENS, is no longer possible. The reason RocksDB
removed NO_FILE_CLOSES seems to be that it did not properly work:
https://github.com/search?q=repo%3Afacebook%2Frocksdb+NO_FIL [...]
- Removal of methods related to compressed block cache configuration in
BlockBasedTableConfig
- Change of the signature of org.rocksdb.Options.setLogger()
Reviewers: Anna Sophie Blee-Goldman <[email protected]>, Matthias J.
Sax <[email protected]>, Bruno Cadonna <[email protected]>
---
LICENSE-binary | 2 +-
docs/ops.html | 2 +-
docs/streams/upgrade-guide.html | 10 ++++
gradle/dependencies.gradle | 2 +-
...tionsToDbOptionsColumnFamilyOptionsAdapter.java | 70 ++++------------------
.../internals/metrics/RocksDBMetricsRecorder.java | 3 +-
...sToDbOptionsColumnFamilyOptionsAdapterTest.java | 6 +-
.../metrics/RocksDBMetricsRecorderTest.java | 8 +--
8 files changed, 31 insertions(+), 72 deletions(-)
diff --git a/LICENSE-binary b/LICENSE-binary
index 8ee1e12da86..c5f65ca15da 100644
--- a/LICENSE-binary
+++ b/LICENSE-binary
@@ -257,7 +257,7 @@ metrics-core-4.1.12.1
metrics-core-2.2.0
opentelemetry-proto-1.0.0-alpha
plexus-utils-3.5.1
-rocksdbjni-7.9.2
+rocksdbjni-9.7.3
scala-library-2.13.15
scala-logging_2.13-3.9.5
scala-reflect-2.13.15
diff --git a/docs/ops.html b/docs/ops.html
index 02ea5bd788f..18a1bf44c5c 100644
--- a/docs/ops.html
+++ b/docs/ops.html
@@ -3475,7 +3475,7 @@ customized state stores; for built-in state stores,
currently we have:
</tr>
<tr>
<td>number-open-files</td>
- <td>The number of current open files.</td>
+ <td>This metric will return constant -1 because the RocksDB's counter
NO_FILE_CLOSES has been removed in RocksDB 9.7.3</td>
<td>kafka.streams:type=stream-state-metrics,thread-id=([-.\w]+),task-id=([-.\w]+),[store-scope]-id=([-.\w]+)</td>
</tr>
<tr>
diff --git a/docs/streams/upgrade-guide.html b/docs/streams/upgrade-guide.html
index 8c1446af7da..8f89224581e 100644
--- a/docs/streams/upgrade-guide.html
+++ b/docs/streams/upgrade-guide.html
@@ -173,6 +173,16 @@
See <a href="https://cwiki.apache.org/confluence/x/TZCMEw";>KIP-1112</a>
for more details.
</p>
+ <p>
+ Upgraded RocksDB dependency to version 9.7.3 (from 7.9.2). This
upgrade incorporates various improvements and optimizations within RocksDB.
However, it also introduces some API changes.
+ The <code>org.rocksdb.AccessHint</code> class, along with its
associated methods, has been removed.
+ Several methods related to compressed block cache configuration in the
<code>BlockBasedTableConfig</code> class have been removed, including
<code>blockCacheCompressedNumShardBits</code>,
<code>blockCacheCompressedSize</code>, and their corresponding setters. These
functionalities are now consolidated under the <code>cache</code> option, and
developers should configure their compressed block cache using the
<code>setCache</code> method instead.
+ The <code>NO_FILE_CLOSES</code> field has been removed from the
<code>org.rocksdb.TickerTypeenum</code> as a result the
<code>number-open-files</code> metrics does not work as expected. Metric
<code>number-open-files</code> returns constant -1 from now on until it will
officially be removed.
+ The <code>org.rocksdb.Options.setLogger()</code> method now accepts a
<code>LoggerInterface</code> as a parameter instead of the previous
<code>Logger</code>.
+ Some data types used in RocksDB's Java API have been modified. These
changes, along with the removed class, field, and new methods, are primarily
relevant to users implementing custom RocksDB configurations.
+ These changes are expected to be largely transparent to most Kafka
Streams users. However, those employing advanced RocksDB customizations within
their Streams applications, particularly through the
<code>rocksdb.config.setter</code>, are advised to consult the detailed RocksDB
9.7.3 changelog to ensure a smooth transition and adapt their configurations as
needed. Specifically, users leveraging the removed <code>AccessHint</code>
class, the removed methods from the <code>BlockBas [...]
+ </p>
+
<h3><a id="streams_api_changes_390"
href="#streams_api_changes_390">Streams API changes in 3.9.0</a></h3>
<p>
diff --git a/gradle/dependencies.gradle b/gradle/dependencies.gradle
index b541de098ed..456bc3d4309 100644
--- a/gradle/dependencies.gradle
+++ b/gradle/dependencies.gradle
@@ -116,7 +116,7 @@ versions += [
protobuf: "3.25.5", // a dependency of opentelemetryProto
pcollections: "4.0.1",
re2j: "1.7",
- rocksDB: "7.9.2",
+ rocksDB: "9.7.3",
// When updating the scalafmt version please also update the version field
in checkstyle/.scalafmt.conf. scalafmt now
// has the version field as mandatory in its configuration, see
// https://github.com/scalameta/scalafmt/releases/tag/v3.1.0.
diff --git
a/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter.java
b/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter.java
index b7b611f8be0..5b2f1e06b34 100644
---
a/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter.java
+++
b/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter.java
@@ -22,7 +22,6 @@ import org.rocksdb.AbstractComparator;
import org.rocksdb.AbstractEventListener;
import org.rocksdb.AbstractSlice;
import org.rocksdb.AbstractWalFilter;
-import org.rocksdb.AccessHint;
import org.rocksdb.BuiltinComparator;
import org.rocksdb.Cache;
import org.rocksdb.ColumnFamilyOptions;
@@ -37,6 +36,7 @@ import org.rocksdb.DBOptions;
import org.rocksdb.DbPath;
import org.rocksdb.Env;
import org.rocksdb.InfoLogLevel;
+import org.rocksdb.LoggerInterface;
import org.rocksdb.MemTableConfig;
import org.rocksdb.MergeOperator;
import org.rocksdb.Options;
@@ -332,14 +332,6 @@ public class
RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter extends
return dbOptions.statistics();
}
- @Deprecated
- public int baseBackgroundCompactions() {
- final String message = "This method has been removed from the
underlying RocksDB. " +
- "It is currently a no-op method which returns a default value
of -1.";
- log.warn(message);
- return -1;
- }
-
@Override
public Options setMaxSubcompactions(final int maxSubcompactions) {
dbOptions.setMaxSubcompactions(maxSubcompactions);
@@ -571,34 +563,6 @@ public class
RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter extends
return dbOptions.dbWriteBufferSize();
}
- @Override
- public Options setAccessHintOnCompactionStart(final AccessHint accessHint)
{
- dbOptions.setAccessHintOnCompactionStart(accessHint);
- return this;
- }
-
- @Override
- public AccessHint accessHintOnCompactionStart() {
- return dbOptions.accessHintOnCompactionStart();
- }
-
- @Deprecated
- public Options setNewTableReaderForCompactionInputs(final boolean
newTableReaderForCompactionInputs) {
- final String message = "This method has been removed from the
underlying RocksDB. " +
- "It was not affecting compaction even in earlier versions. " +
- "It is currently a no-op method.";
- log.warn(message);
- return this;
- }
-
- @Deprecated
- public boolean newTableReaderForCompactionInputs() {
- final String message = "This method has been removed from the
underlying RocksDB. " +
- "It is now a method which always returns false.";
- log.warn(message);
- return false;
- }
-
@Override
public Options setCompactionReadaheadSize(final long
compactionReadaheadSize) {
dbOptions.setCompactionReadaheadSize(compactionReadaheadSize);
@@ -843,7 +807,7 @@ public class
RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter extends
}
@Override
- public Options setLogger(final org.rocksdb.Logger logger) {
+ public Options setLogger(final LoggerInterface logger) {
dbOptions.setLogger(logger);
return this;
}
@@ -914,6 +878,16 @@ public class
RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter extends
return this;
}
+ @Override
+ public Options setMemtableMaxRangeDeletions(final int n) {
+ columnFamilyOptions.setMemtableMaxRangeDeletions(n);
+ return this;
+ }
+
+ @Override
+ public int memtableMaxRangeDeletions() {
+ return columnFamilyOptions.memtableMaxRangeDeletions();
+ }
@Override
public Options setBottommostCompressionType(final CompressionType
bottommostCompressionType) {
@@ -1464,26 +1438,6 @@ public class
RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter extends
return dbOptions.allowIngestBehind();
}
- @Deprecated
- public Options setPreserveDeletes(final boolean preserveDeletes) {
- final String message = "This method has been removed from the
underlying RocksDB. " +
- "It was marked for deprecation in earlier versions. " +
- "The behaviour can be replicated by using user-defined
timestamps. " +
- "It is currently a no-op method.";
- log.warn(message);
- // no-op
- return this;
- }
-
- @Deprecated
- public boolean preserveDeletes() {
- final String message = "This method has been removed from the
underlying RocksDB. " +
- "It was marked for deprecation in earlier versions. " +
- "It is currently a no-op method with a default value of
false.";
- log.warn(message);
- return false;
- }
-
@Override
public Options setTwoWriteQueues(final boolean twoWriteQueues) {
dbOptions.setTwoWriteQueues(twoWriteQueues);
diff --git
a/streams/src/main/java/org/apache/kafka/streams/state/internals/metrics/RocksDBMetricsRecorder.java
b/streams/src/main/java/org/apache/kafka/streams/state/internals/metrics/RocksDBMetricsRecorder.java
index 03b1f7eaf02..10e8cb804fe 100644
---
a/streams/src/main/java/org/apache/kafka/streams/state/internals/metrics/RocksDBMetricsRecorder.java
+++
b/streams/src/main/java/org/apache/kafka/streams/state/internals/metrics/RocksDBMetricsRecorder.java
@@ -462,8 +462,7 @@ public class RocksDBMetricsRecorder {
writeStallDuration +=
valueProviders.statistics.getAndResetTickerCount(TickerType.STALL_MICROS);
bytesWrittenDuringCompaction +=
valueProviders.statistics.getAndResetTickerCount(TickerType.COMPACT_WRITE_BYTES);
bytesReadDuringCompaction +=
valueProviders.statistics.getAndResetTickerCount(TickerType.COMPACT_READ_BYTES);
- numberOfOpenFiles +=
valueProviders.statistics.getAndResetTickerCount(TickerType.NO_FILE_OPENS)
- -
valueProviders.statistics.getAndResetTickerCount(TickerType.NO_FILE_CLOSES);
+ numberOfOpenFiles = -1;
numberOfFileErrors +=
valueProviders.statistics.getAndResetTickerCount(TickerType.NO_FILE_ERRORS);
final HistogramData memtableFlushTimeData =
valueProviders.statistics.getHistogramData(HistogramType.FLUSH_TIME);
memtableFlushTimeSum += memtableFlushTimeData.getSum();
diff --git
a/streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.java
b/streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.java
index 5ddcf5bef55..08248b02054 100644
---
a/streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.java
+++
b/streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.java
@@ -28,7 +28,6 @@ import org.rocksdb.AbstractCompactionFilter;
import org.rocksdb.AbstractCompactionFilter.Context;
import org.rocksdb.AbstractCompactionFilterFactory;
import org.rocksdb.AbstractWalFilter;
-import org.rocksdb.AccessHint;
import org.rocksdb.BuiltinComparator;
import org.rocksdb.ColumnFamilyOptions;
import org.rocksdb.CompactionPriority;
@@ -112,6 +111,8 @@ public class
RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest {
add("setMaxBackgroundCompactions");
add("maxBackgroundFlushes");
add("setMaxBackgroundFlushes");
+ add("tablePropertiesCollectorFactory");
+ add("setTablePropertiesCollectorFactory");
addAll(walRelatedMethods);
}
};
@@ -176,9 +177,6 @@ public class
RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest {
case "java.util.Collection":
parameters[i] = new ArrayList<>();
break;
- case "org.rocksdb.AccessHint":
- parameters[i] = AccessHint.NONE;
- break;
case "org.rocksdb.Cache":
parameters[i] = new LRUCache(1L);
break;
diff --git
a/streams/src/test/java/org/apache/kafka/streams/state/internals/metrics/RocksDBMetricsRecorderTest.java
b/streams/src/test/java/org/apache/kafka/streams/state/internals/metrics/RocksDBMetricsRecorderTest.java
index 7ec3f4bf38c..a0c068b59ee 100644
---
a/streams/src/test/java/org/apache/kafka/streams/state/internals/metrics/RocksDBMetricsRecorderTest.java
+++
b/streams/src/test/java/org/apache/kafka/streams/state/internals/metrics/RocksDBMetricsRecorderTest.java
@@ -474,10 +474,8 @@ public class RocksDBMetricsRecorderTest {
final double expectedCompactionTimeMaxSensor = 24.0;
when(statisticsToAdd1.getAndResetTickerCount(TickerType.NO_FILE_OPENS)).thenReturn(5L);
-
when(statisticsToAdd1.getAndResetTickerCount(TickerType.NO_FILE_CLOSES)).thenReturn(3L);
when(statisticsToAdd2.getAndResetTickerCount(TickerType.NO_FILE_OPENS)).thenReturn(7L);
-
when(statisticsToAdd2.getAndResetTickerCount(TickerType.NO_FILE_CLOSES)).thenReturn(4L);
- final double expectedNumberOfOpenFilesSensor = (5 + 7) - (3 + 4);
+ final double expectedNumberOfOpenFilesSensor = -1;
when(statisticsToAdd1.getAndResetTickerCount(TickerType.NO_FILE_ERRORS)).thenReturn(34L);
when(statisticsToAdd2.getAndResetTickerCount(TickerType.NO_FILE_ERRORS)).thenReturn(11L);
@@ -485,8 +483,8 @@ public class RocksDBMetricsRecorderTest {
recorder.record(now);
- verify(statisticsToAdd1,
times(17)).getAndResetTickerCount(isA(TickerType.class));
- verify(statisticsToAdd2,
times(17)).getAndResetTickerCount(isA(TickerType.class));
+ verify(statisticsToAdd1,
times(15)).getAndResetTickerCount(isA(TickerType.class));
+ verify(statisticsToAdd2,
times(15)).getAndResetTickerCount(isA(TickerType.class));
verify(statisticsToAdd1,
times(2)).getHistogramData(isA(HistogramType.class));
verify(statisticsToAdd2,
times(2)).getHistogramData(isA(HistogramType.class));
verify(bytesWrittenToDatabaseSensor).record(expectedBytesWrittenToDatabaseSensor,
now);
