Mark Cho created FLINK-11133:
--------------------------------
Summary: FsCheckpointStorage is unaware about S3 entropy when
creating directories
Key: FLINK-11133
URL: https://issues.apache.org/jira/browse/FLINK-11133
Project: Flink
Issue Type: Bug
Affects Versions: 1.7.0
Reporter: Mark Cho
We currently use S3 for our checkpoint storage with S3 entropy enabled.
Entropy seems to be working correctly when writing out checkpoint metadata file
(entropy key is correctly stripped from `state.checkpoints.dir`) and when
writing out checkpoint data file (entropy key is correctly replaced with random
string).
However, from the logs, it seems like entropy key is not stripped or replaced
when `FsCheckpointStorage` creates directories in the following class:
[https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/state/filesystem/FsCheckpointStorage.java#L83-L85]
Should FsCheckpointStorage skip initializing mkdir calls if object store like
S3 is used since S3 doesn't have directory concept?
If we want to keep the `mkdir` calls in `FsCheckpointStorage`, we should handle
the entropy key specified in `state.checkpoints.dir`. Currently, folder markers
in S3 are being created by Hadoop FileSystem with the entropy key in the path
as a result of `mkdir` calls in `FsCheckpointStorage`).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
