klion26 commented on a change in pull request #7351: [FLINK-11008][State
Backends, Checkpointing]SpeedUp upload state files using multithread
URL: https://github.com/apache/flink/pull/7351#discussion_r246358163
##########
File path:
flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDbStateDataTransfer.java
##########
@@ -61,6 +69,88 @@ static void transferAllStateDataToDirectory(
downloadDataForAllStateHandles(miscFiles, dest,
restoringThreadNum, closeableRegistry);
}
+ public static void uploadFilesToCheckpointFs(
+ @Nonnull Map<StateHandleID, Path> files,
+ int numberOfSnapshottingThreads,
+ CheckpointStreamFactory checkpointStreamFactory,
+ CloseableRegistry closeableRegistry,
+ Map<StateHandleID, StreamStateHandle> hanldes) throws Exception
{
Review comment:
@azagrebin The implementation here returns `handles` implicitly because I
thought the caller of the function `uploadFilesToCheckpointFs` may know the
size of `handles`, and could init the map by `new HashMap(size)`.
I agree that the map's size here would not be too large because we'are under
incremental mode, I'll change to the explicitly mode. Anyway, I'll add a java
doc for these two public function.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
