HuangZhenQiu commented on code in PR #26640:
URL: https://github.com/apache/flink/pull/26640#discussion_r2155113879
##########
docs/layouts/shortcodes/generated/expert_rocksdb_section.html:
##########
@@ -14,6 +14,12 @@
<td>Integer</td>
<td>The number of threads (per stateful operator) used to transfer
(download and upload) files in RocksDBStateBackend.If negative, the common (TM)
IO thread pool is used (see cluster.io-pool.size)</td>
</tr>
+ <tr>
+ <td><h5>state.backend.rocksdb.checkpoint.upload-jitter</h5></td>
+ <td style="word-wrap: break-word;">0 ms</td>
+ <td>Duration</td>
+ <td>The time interval used to create jitter for each checkpoint
file upload.</td>
Review Comment:
Thanks. It is a good question. The state uploader is in async path. The
jitter help to span the upload request with in a time interval to reduce the
spike traffic to cloud/remote storage.
1. Whether there is a way to check for the rate limitting return code and
only delay those.
From our observation, the IO exception will throw within file system layer.
In this case, Flink with fail current checkpoint. There is no easy way to use
return code to adjust the jitter. We probably may add exponential delayed retry
after catch IOException. It is out of our original scope of the problem we want
to resolve. But it could be a further improve in another PR.
2. if we add a random delay using the Jitter , is there a case that the
checkpoints might end up out of order?
How many checkpoints can happen is controlled by the allowed concurrent
checkpoints configured by users. The jitter will add additional time for each
task manager to finish the checkpoint and send back the ack to checkpoint
coordinator. Given checkpoint interval is usually configured in minutes level,
and the jitter is configured in seconds level. I don't the risk of checkpoint
out of order.
@mxm What's your opinion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
