This is an automated email from the ASF dual-hosted git repository.
srichter pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit 6bf97674b660dbe1cd96b5b0273c3954f2c16fba
Author: Stefan Richter <srich...@confluent.io>
AuthorDate: Mon Jan 22 15:50:09 2024 +0100
[FLINK-34199] Add documentation.
---
docs/content.zh/docs/ops/traces.md | 14 +++++++++++---
docs/content/docs/ops/traces.md | 14 +++++++++++---
.../generated/rocksdb_configurable_configuration.html | 12 ++++++++++++
3 files changed, 34 insertions(+), 6 deletions(-)
diff --git a/docs/content.zh/docs/ops/traces.md
b/docs/content.zh/docs/ops/traces.md
index d5025b44d02..8157d1ecddd 100644
--- a/docs/content.zh/docs/ops/traces.md
+++ b/docs/content.zh/docs/ops/traces.md
@@ -97,7 +97,7 @@ Flink reports a single span trace for the whole checkpoint
and job initializatio
</thead>
<tbody>
<tr>
- <th
rowspan="16">org.apache.flink.</br>runtime.checkpoint.</br>CheckpointStatsTracker</th>
+ <th
rowspan="18">org.apache.flink.</br>runtime.checkpoint.</br>CheckpointStatsTracker</th>
<th rowspan="6"><strong>Checkpoint</strong></th>
<td>startTs</td>
<td>Timestamp when the checkpoint has started.</td>
@@ -123,7 +123,7 @@ Flink reports a single span trace for the whole checkpoint
and job initializatio
<td>What was the state of this checkpoint: FAILED or COMPLETED.</td>
</tr>
<tr>
- <th rowspan="10"><strong>JobInitialization</strong></th>
+ <th rowspan="12"><strong>JobInitialization</strong></th>
<td>startTs</td>
<td>Timestamp when the job initialization has started.</td>
</tr>
@@ -157,7 +157,11 @@ Flink reports a single span trace for the whole checkpoint
and job initializatio
</tr>
<tr>
<td>(Max/Sum)DownloadStateDurationMs<br><br>(optional - currently only
supported by RocksDB Incremental)</td>
- <td>The aggregated (max and sum) across all subtasks duration of
downloading state files from the DFS.</td>
+ <td>The aggregated (max and sum) duration across all subtasks of
downloading state files from the DFS.</td>
+ </tr>
+ <tr>
+ <td>(Max/Sum)RestoreStateDurationMs<br><br>(optional - currently only
supported by RocksDB Incremental)</td>
+ <td>The aggregated (max and sum) duration across all subtasks of
restoring the state backend from fully localized state, i.e. after all remote
state was downloaded.</td>
</tr>
<tr>
<td>(Max/Sum)RestoredStateSizeBytes.[location]</td>
@@ -167,6 +171,10 @@ Flink reports a single span trace for the whole checkpoint
and job initializatio
REMOTE,
UNKNOWN.</td>
</tr>
+ <tr>
+ <td>(Max/Sum)RestoreAsyncCompactionDurationMs<br><br>(optional -
currently only supported by RocksDB Incremental)</td>
+ <td>The aggregated (max and sum) duration across all subtasks for async
compaction after incremental restore.</td>
+ </tr>
</tbody>
</table>
diff --git a/docs/content/docs/ops/traces.md b/docs/content/docs/ops/traces.md
index d5025b44d02..8157d1ecddd 100644
--- a/docs/content/docs/ops/traces.md
+++ b/docs/content/docs/ops/traces.md
@@ -97,7 +97,7 @@ Flink reports a single span trace for the whole checkpoint
and job initializatio
</thead>
<tbody>
<tr>
- <th
rowspan="16">org.apache.flink.</br>runtime.checkpoint.</br>CheckpointStatsTracker</th>
+ <th
rowspan="18">org.apache.flink.</br>runtime.checkpoint.</br>CheckpointStatsTracker</th>
<th rowspan="6"><strong>Checkpoint</strong></th>
<td>startTs</td>
<td>Timestamp when the checkpoint has started.</td>
@@ -123,7 +123,7 @@ Flink reports a single span trace for the whole checkpoint
and job initializatio
<td>What was the state of this checkpoint: FAILED or COMPLETED.</td>
</tr>
<tr>
- <th rowspan="10"><strong>JobInitialization</strong></th>
+ <th rowspan="12"><strong>JobInitialization</strong></th>
<td>startTs</td>
<td>Timestamp when the job initialization has started.</td>
</tr>
@@ -157,7 +157,11 @@ Flink reports a single span trace for the whole checkpoint
and job initializatio
</tr>
<tr>
<td>(Max/Sum)DownloadStateDurationMs<br><br>(optional - currently only
supported by RocksDB Incremental)</td>
- <td>The aggregated (max and sum) across all subtasks duration of
downloading state files from the DFS.</td>
+ <td>The aggregated (max and sum) duration across all subtasks of
downloading state files from the DFS.</td>
+ </tr>
+ <tr>
+ <td>(Max/Sum)RestoreStateDurationMs<br><br>(optional - currently only
supported by RocksDB Incremental)</td>
+ <td>The aggregated (max and sum) duration across all subtasks of
restoring the state backend from fully localized state, i.e. after all remote
state was downloaded.</td>
</tr>
<tr>
<td>(Max/Sum)RestoredStateSizeBytes.[location]</td>
@@ -167,6 +171,10 @@ Flink reports a single span trace for the whole checkpoint
and job initializatio
REMOTE,
UNKNOWN.</td>
</tr>
+ <tr>
+ <td>(Max/Sum)RestoreAsyncCompactionDurationMs<br><br>(optional -
currently only supported by RocksDB Incremental)</td>
+ <td>The aggregated (max and sum) duration across all subtasks for async
compaction after incremental restore.</td>
+ </tr>
</tbody>
</table>
diff --git
a/docs/layouts/shortcodes/generated/rocksdb_configurable_configuration.html
b/docs/layouts/shortcodes/generated/rocksdb_configurable_configuration.html
index 8810b9086d1..4ec59a27110 100644
--- a/docs/layouts/shortcodes/generated/rocksdb_configurable_configuration.html
+++ b/docs/layouts/shortcodes/generated/rocksdb_configurable_configuration.html
@@ -74,6 +74,12 @@
<td>Integer</td>
<td>The maximum number of open files (per stateful operator) that
can be used by the DB, '-1' means no limit. The default value is '-1'.</td>
</tr>
+ <tr>
+
<td><h5>state.backend.rocksdb.incremental-restore-async-compact-after-rescale</h5></td>
+ <td style="word-wrap: break-word;">false</td>
+ <td>Boolean</td>
+ <td>If true, an async compaction of RocksDB is started after every
restore after which we detect keys (including tombstones) in the database that
are outside the key-groups range of the backend.</td>
+ </tr>
<tr>
<td><h5>state.backend.rocksdb.log.dir</h5></td>
<td style="word-wrap: break-word;">(none)</td>
@@ -116,6 +122,12 @@
<td>Boolean</td>
<td>If true, every newly created SST file will contain a Bloom
filter. It is disabled by default.</td>
</tr>
+ <tr>
+ <td><h5>state.backend.rocksdb.use-ingest-db-restore-mode</h5></td>
+ <td style="word-wrap: break-word;">false</td>
+ <td>Boolean</td>
+ <td>A recovery mode that directly clips and ingests multiple DBs
during state recovery if the keys in the SST files does not exceed the declared
key-group range.</td>
+ </tr>
<tr>
<td><h5>state.backend.rocksdb.write-batch-size</h5></td>
<td style="word-wrap: break-word;">2 mb</td>
