[
https://issues.apache.org/jira/browse/FLINK-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-11133:
-----------------------------------
Component/s: FileSystems
> FsCheckpointStorage is unaware about S3 entropy when creating directories
> -------------------------------------------------------------------------
>
> Key: FLINK-11133
> URL: https://issues.apache.org/jira/browse/FLINK-11133
> Project: Flink
> Issue Type: Bug
> Components: FileSystems
> Affects Versions: 1.7.0
> Reporter: Mark Cho
> Priority: Minor
>
> We currently use S3 for our checkpoint storage with S3 entropy enabled.
> Entropy seems to be working correctly when writing out checkpoint metadata
> file (entropy key is correctly stripped from `state.checkpoints.dir`) and
> when writing out checkpoint data file (entropy key is correctly replaced with
> random string).
> However, from the logs, it seems like entropy key is not stripped or replaced
> when `FsCheckpointStorage` creates directories in the following class:
> [https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/state/filesystem/FsCheckpointStorage.java#L83-L85]
> Should FsCheckpointStorage skip initializing mkdir calls if object store like
> S3 is used since S3 doesn't have directory concept?
> If we want to keep the `mkdir` calls in `FsCheckpointStorage`, we should
> handle the entropy key specified in `state.checkpoints.dir`. Currently,
> folder markers in S3 are being created by Hadoop FileSystem with the entropy
> key in the path as a result of `mkdir` calls in `FsCheckpointStorage`).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
