StefanRRichter commented on a change in pull request #8263:
[FLINK-12296][StateBackend]Data loss silently in RocksDBStateBackend because of
local directory collision
URL: https://github.com/apache/flink/pull/8263#discussion_r279276307
##########
File path:
flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/snapshot/RocksIncrementalSnapshotStrategy.java
##########
@@ -184,17 +189,19 @@ private SnapshotDirectory
prepareLocalSnapshotDirectory(long checkpointId) throw
LocalRecoveryDirectoryProvider directoryProvider =
localRecoveryConfig.getLocalStateDirectoryProvider();
File directory =
directoryProvider.subtaskSpecificCheckpointDirectory(checkpointId);
- if (directory.exists()) {
- FileUtils.deleteDirectory(directory);
- }
-
- if (!directory.mkdirs()) {
+ if (!directory.exists() && !directory.mkdirs()) {
throw new IOException("Local state base
directory for checkpoint " + checkpointId +
" already exists: " + directory);
}
// introduces an extra directory because RocksDB wants
a non-existing directory for native checkpoints.
- File rdbSnapshotDir = new File(directory, "rocks_db");
+ // append operatorIdentifier here to solve directory
collision problem when two stateful operators chained in one task.
+ String subDir =
operatorIdentifier.replaceAll("[^a-zA-Z0-9\\-]", "_");
Review comment:
I would suggest to move the string sanatizing into a separate method and we
could use a precompiled regex that we can reuse every time over `replaceAll`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
