azagrebin commented on a change in pull request #7351: [FLINK-11008][State
Backends, Checkpointing]SpeedUp upload state files using multithread
URL: https://github.com/apache/flink/pull/7351#discussion_r246444142
##########
File path:
flink-state-backends/flink-statebackend-rocksdb/src/test/java/org/apache/flink/contrib/streaming/state/RocksDBStateDataTransferTest.java
##########
@@ -138,6 +149,133 @@ public void testMultiThreadRestoreCorrectly() throws
Exception {
}
}
+ /**
+ * Test that the exception arose in the thread pool will rethrow to the
main thread.
+ */
+ @Test
+ public void testMultiThreadUploadThreadPoolExceptionRethrow() throws
IOException {
+ SpecifiedException expectedException = new
SpecifiedException("throw exception while multi thread upload states.");
+
+ CheckpointStateOutputStream outputStream =
createFailingCheckpointStateOutputStream(expectedException);
+ CheckpointStreamFactory checkpointStreamFactory =
(CheckpointedStateScope scope) -> outputStream;
+
+ File file =
temporaryFolder.newFile(String.valueOf(UUID.randomUUID()));
+ generateRandomFileContent(file.getPath(), 20);
+
+ Map<StateHandleID, Path> filePaths = new HashMap<>(1);
+ filePaths.put(new StateHandleID("mockHandleID"), new
Path(file.getPath()));
+ try {
+ RocksDbStateDataTransfer.uploadFilesToCheckpointFs(
+ filePaths,
+ 5,
+ checkpointStreamFactory,
+ new CloseableRegistry());
+ fail();
+ } catch (Exception e) {
+ assertEquals(expectedException, e);
+ }
+ }
+
+ /**
+ * Test that upload files with multi-thread correctly.
+ */
+ @Test
+ public void testMultiThreadUploadCorrectly() throws Exception {
+
+ File checkpointPrivateFolder =
temporaryFolder.newFolder("private");
+ Path checkpointPrivateDirectory = new
Path(checkpointPrivateFolder.getPath());
+
+ File checkpointSharedFolder =
temporaryFolder.newFolder("shared");
+ Path checkpointSharedDirectory = new
Path(checkpointSharedFolder.getPath());
+
+ FileSystem fileSystem =
checkpointPrivateDirectory.getFileSystem();
+ int fileStateSizeThreshold = 1024;
+ FsCheckpointStreamFactory checkpointStreamFactory =
+ new FsCheckpointStreamFactory(fileSystem,
checkpointPrivateDirectory, checkpointSharedDirectory, fileStateSizeThreshold);
+
+ String localFolder = "local";
+ temporaryFolder.newFolder(localFolder);
+
+ int sstFileCount = 6;
+ Map<StateHandleID, Path> sstFilePaths =
generateRandomSstFiles(localFolder, sstFileCount, fileStateSizeThreshold);
+
+ Map<StateHandleID, StreamStateHandle> sstFiles = new
HashMap<>(sstFileCount);
+
+
sstFiles.putAll(RocksDbStateDataTransfer.uploadFilesToCheckpointFs(
Review comment:
I think it can be directly assigned to
`Map<StateHandleID, StreamStateHandle> sstFiles =
uploadFilesToCheckpointFs(..)`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
