[
https://issues.apache.org/jira/browse/FLINK-24878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17442444#comment-17442444
]
Jun Qin edited comment on FLINK-24878 at 11/11/21, 8:26 PM:
------------------------------------------------------------
>From the doc:
{quote}The Flink runtime currently passes the option to inject entropy only to
checkpoint data files. All other files, including checkpoint metadata and
external URI, do not inject entropy to keep checkpoint URIs predictable.
{quote}
[~danderson] Is your checkpoint large enough such that it has some data files
in addition to the metadata file? If so, do you see some files created under
{{s3p://alpinegizmo-test-bucket/xxxxxx}} in your test?
was (Author: qinjunjerry):
>From the doc:
{quote}The Flink runtime currently passes the option to inject entropy only to
checkpoint data files. All other files, including checkpoint metadata and
external URI, do not inject entropy to keep checkpoint URIs predictable.
{quote}
[~danderson] If your checkpoint large enough such that it has some data files
in addition to the metadata file? If so, do you see some files created under
{{s3p://alpinegizmo-test-bucket/xxxxxx}} in your test?
> Flink entropy injection not working
> -----------------------------------
>
> Key: FLINK-24878
> URL: https://issues.apache.org/jira/browse/FLINK-24878
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
> Affects Versions: 1.11.4, 1.12.0, 1.14.0
> Reporter: Kenny Ma
> Priority: Major
>
> I've followed the instructions described by this doc
> [https://nightlies.apache.org/flink/flink-docs-release-1.11/ops/filesystems/s3.html]
> to enable entropy injection for Flink checkpointing. Unfortunately the
> feature is not working and the entropy key is not being replaced in the AWS
> S3 checkpoint location.
> To reproduce the problem with the Flink StateMachineExample example:
> {code:java}
> $FLINK_HOME/bin/flink run -c
> org.apache.flink.streaming.examples.statemachine.StateMachineExample
> examples/streaming/StateMachineExample.jar --sleep 100 --backend file
> --checkpoint-dir s3://s3-bucket-xyz/_entropy_/checkpoint/
> --incremental-checkpoints true{code}
>
> The behavior is different depending on the S3 filesystem library used.
> If using the flink-s3-fs-hadoop library, the S3 path for the checkpoint is
> s3://s3-bucket-xyz/_entropy_/checkpoint/...
> If using the flink-s3-fs-presto library, the S3 path for the checkpoint is
> s3://s3-bucket-xyz/checkpoint/...
> For both S3 filesystem libraries the entropy injection failed to happen.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
