carp84 commented on a change in pull request #10987: [FLINK-14495][docs]
(EXTENDED) Add documentation for memory control of RocksDB state backend
URL: https://github.com/apache/flink/pull/10987#discussion_r373755840
##########
File path: docs/ops/state/large_state_tuning.md
##########
@@ -118,98 +118,70 @@ Other state like keyed state is still snapshotted
asynchronously. Please note th
The state storage workhorse of many large scale Flink streaming applications
is the *RocksDB State Backend*.
The backend scales well beyond main memory and reliably stores large [keyed
state](../../dev/stream/state/state.html).
-Unfortunately, RocksDB's performance can vary with configuration, and there is
little documentation on how to tune
-RocksDB properly. For example, the default configuration is tailored towards
SSDs and performs suboptimal
-on spinning disks.
+RocksDB's performance can vary with configuration, this section outlines some
best-practices for tuning jobs that use the RocksDB State Backend.
-**Incremental Checkpoints**
+### Incremental Checkpoints
-Incremental checkpoints can dramatically reduce the checkpointing time in
comparison to full checkpoints, at the cost of a (potentially) longer
-recovery time. The core idea is that incremental checkpoints only record all
changes to the previous completed checkpoint, instead of
-producing a full, self-contained backup of the state backend. Like this,
incremental checkpoints build upon previous checkpoints. Flink leverages
-RocksDB's internal backup mechanism in a way that is self-consolidating over
time. As a result, the incremental checkpoint history in Flink
-does not grow indefinitely, and old checkpoints are eventually subsumed and
pruned automatically.
+When it comes to reducing the time that checkpoints take, activating
incremental checkpoints should be one of the first considerations.
+Incremental checkpoints can dramatically reduce the checkpointing time in
comparison to full checkpoints, because incremental checkpoints only record the
changes compared to the previous completed checkpoint, instead of producing a
full, self-contained backup of the state backend.
-While we strongly encourage the use of incremental checkpoints for large
state, please note that this is a new feature and currently not enabled
-by default. To enable this feature, users can instantiate a
`RocksDBStateBackend` with the corresponding boolean flag in the constructor
set to `true`, e.g.:
+See [Incremental Checkpoints in RocksDB]({{ site.baseurl
}}/ops/state/state_backends.html#incremental-checkpoints) for more background
information.
-{% highlight java %}
- RocksDBStateBackend backend =
- new RocksDBStateBackend(filebackend, true);
-{% endhighlight %}
-
-**RocksDB Timers**
+### Timers in RocksDB or on JVM Heap
-For RocksDB, a user can chose whether timers are stored on the heap or inside
RocksDB (default). Heap-based timers can have a better performance for smaller
numbers of
-timers, while storing timers inside RocksDB offers higher scalability as the
number of timers in RocksDB can exceed the available main memory (spilling to
disk).
+Timers are stored in RocksDB by default, which is the more robust and scalable
choice.
-When using RockDB as state backend, the type of timer storage can be selected
through Flink's configuration via option key
`state.backend.rocksdb.timer-service.factory`.
-Possible choices are `heap` (to store timers on the heap, default) and
`rocksdb` (to store timers in RocksDB).
+When performance-tuning jobs that have few timers only (no windows, not using
timers in ProcessFunction), putting those timers on the heap can increase
performance.
+Use this feature carefully, as heap-based timers may increase checkpointing
times and naturally cannot scale beyond memory.
-<span class="label label-info">Note</span> *The combination RocksDB state
backend with heap-based timers currently does NOT support asynchronous
snapshots for the timers state.
-Other state like keyed state is still snapshotted asynchronously. Please note
that this is not a regression from previous versions and will be resolved with
`FLINK-10026`.*
+See [this section]({{ site.baseurl
}}/ops/state/state_backends.html#timers-heap-vs-rocksdb) for details how to
configure heap-based timers.
Review comment:
for details **_of_** how to configure
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
