[
https://issues.apache.org/jira/browse/FLINK-27967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17555535#comment-17555535
]
Prabhu Joseph commented on FLINK-27967:
---------------------------------------
We identified this to be a configuration issue. Flink supports entropy
injection only when FlinkS3FileSystem is used and not when Hadoop S3FileSystem
is used. Adding below in flink-conf.yaml has fixed the issue
{code}
yarn.ship-files: /usr/lib/flink/opt/flink-s3-fs-hadoop-1.14.0.jar
#fs.allowed-fallback-filesystems: s3
{code}
> Unable to inject entropy on s3 prefix when doing a save point in Flink
> ----------------------------------------------------------------------
>
> Key: FLINK-27967
> URL: https://issues.apache.org/jira/browse/FLINK-27967
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
> Affects Versions: 1.14.0
> Reporter: Vinay Devadiga
> Priority: Major
> Attachments: Screenshot 2022-06-09 at 12.10.51 PM.png
>
>
> Hi while using flink 1.14.0,I ran the example job
> /examples/streaming/StateMachineExample.jar and the job submitted
> successfully and then I tried to save point it to s3 with entropy enabled but
> the entropy key was not respected here are my configurations.Can anyone
> please guide me through issue, I an trying to go through the code but could
> not find anything sustainable.
> |flink-conf|fs.allowed-fallback-filesystems|s3|Cluster configuration| |
> |flink-conf|execution.checkpointing.unaligned|true|Cluster configuration| |
> |flink-conf|state.backend.incremental|true|Cluster configuration| |
> |flink-conf|execution.checkpointing.timeout|600min|Cluster configuration| |
> |flink-conf|execution.checkpointing.externalized-checkpoint-retention|RETAIN_ON_CANCELLATION|Cluster
> configuration| |
> |flink-conf|state.backend|rocksdb|Cluster configuration| |
> |flink-conf|s3.entropy.key|_entropy_|Cluster configuration| |
> |flink-conf|state.checkpoints.dir|s3://vinaydevuswest2/_entropy_/flink/checkpoint-data/|Cluster
> configuration| |
> |flink-conf|execution.checkpointing.max-concurrent-checkpoints|1|Cluster
> configuration| |
> |flink-conf|execution.checkpointing.min-pause|5000|Cluster configuration| |
> |flink-conf|execution.checkpointing.checkpoints-after-tasks-finish.enabled|true|Cluster
> configuration| |
> |flink-conf|state.savepoints.dir|s3://vinaydevuswest2/_entropy_/flink/savepoint-data/|Cluster
> configuration| |
> |flink-conf|state.storage.fs.memory-threshold|0|Cluster configuration| |
> |flink-conf|s3.entropy.length|4|Cluster configuration| |
> |flink-conf|execution.checkpointing.tolerable-failed-checkpoints|30|Cluster
> configuration| |
> |flink-conf|execution.checkpointing.mode|EXACTLY_ONCE|Cluster configuration|
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
