[
https://issues.apache.org/jira/browse/FLINK-24159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Feifan Wang updated FLINK-24159:
--------------------------------
Description:
FLINK-9061 incroduce entropy inject to s3 path for better scalability, but in
document of
[entropy-injection-for-s3-file-systems|https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/filesystems/s3/#entropy-injection-for-s3-file-systems]
use a example with checkpoint directory
"{color:#ff0000}s3://my-bucket/checkpoints/_entropy_/dashboard-job/{color}",
with this configuration every checkpoint key will still start with constant
checkpoints/ prefix which actually reduces scalability.
Thanks to dmtolpeko for describing this issue in his blog (
[flink-and-s3-entropy-injection-for-checkpoints
|http://cloudsqale.com/2021/01/02/flink-and-s3-entropy-injection-for-checkpoints/]).
h3. Proposal
alter the checkpoint directory in document of
[entropy-injection-for-s3-file-systems|https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/filesystems/s3/#entropy-injection-for-s3-file-systems]
to "{color:#ff0000}s3://my-bucket/_entropy_/checkpoints/dashboard-job/{color}"
(make entropy key at start of keys).
If this proposal is appropriate, I am glad to submit a PR to modify the
document here. Any other ideas for this ?
was:
FLINK-9061 incroduce entropy inject to s3 path for better scalability, but in
document of
[entropy-injection-for-s3-file-systems|https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/filesystems/s3/#entropy-injection-for-s3-file-systems]
use a example with checkpoint directory
"{color:#FF0000}s3://my-bucket/checkpoints/_entropy_/dashboard-job/{color}",
with this configuration every checkpoint key will still start with constant
checkpoints/ prefix which actually reduces scalability.
Thanks to dmtolpeko for describing this issue in his blog (
[flink-and-s3-entropy-injection-for-checkpoints
|http://cloudsqale.com/2021/01/02/flink-and-s3-entropy-injection-for-checkpoints/]).
h3. Proposal
alter the checkpoint directory in document of
[entropy-injection-for-s3-file-systems|https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/filesystems/s3/#entropy-injection-for-s3-file-systems]
to "{color:#FF0000}s3://my-bucket/_entropy_/checkpoints/dashboard-job/{color}"
(make entropy key at start of keys).
If this proposal is appropriate, I am glad to submit a PR to modify the
document here. Any other ideas for this ?
> document of entropy injection may mislead users
> -----------------------------------------------
>
> Key: FLINK-24159
> URL: https://issues.apache.org/jira/browse/FLINK-24159
> Project: Flink
> Issue Type: Improvement
> Components: Documentation, Runtime / Checkpointing
> Reporter: Feifan Wang
> Priority: Major
>
> FLINK-9061 incroduce entropy inject to s3 path for better scalability, but in
> document of
> [entropy-injection-for-s3-file-systems|https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/filesystems/s3/#entropy-injection-for-s3-file-systems]
> use a example with checkpoint directory
> "{color:#ff0000}s3://my-bucket/checkpoints/_entropy_/dashboard-job/{color}",
> with this configuration every checkpoint key will still start with constant
> checkpoints/ prefix which actually reduces scalability.
> Thanks to dmtolpeko for describing this issue in his blog (
> [flink-and-s3-entropy-injection-for-checkpoints
> |http://cloudsqale.com/2021/01/02/flink-and-s3-entropy-injection-for-checkpoints/]).
> h3. Proposal
> alter the checkpoint directory in document of
> [entropy-injection-for-s3-file-systems|https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/filesystems/s3/#entropy-injection-for-s3-file-systems]
> to
> "{color:#ff0000}s3://my-bucket/_entropy_/checkpoints/dashboard-job/{color}"
> (make entropy key at start of keys).
>
> If this proposal is appropriate, I am glad to submit a PR to modify the
> document here. Any other ideas for this ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
