GitHub user indrc opened a pull request:
https://github.com/apache/flink/pull/6302
[FLINK-9061][checkpointing] add entropy to s3 path for better scalability
## What is the purpose of the change
*This pull request adds entropy to the checkpoint path based upon a user
defined pattern in the configuration for better s3 scalability*
## Brief change log
- *Read the entropy pattern from the config*
- *In the checkpoint path for FSbackend, substitute the entropy key with
a 4 character random alpha numeric string*
## Verifying this change
This change added tests and can be verified as follows:
- *Added unit tests under FileStateBackendTest to cover different
scenarios for the entropy substitution*
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes )
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: ( no)
- The serializers: ( no )
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes )
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/indrc/flink entropy
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/6302.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6302
----
commit 96def47c57ae58f9717ab35d41e1a4bf6d152851
Author: Indrajit Roychoudhury <iroychoudhury@...>
Date: 2018-06-26T06:36:49Z
[FLINK-9061][checkpointing] add entropy to s3 path for better scalability
commit 15c06fa70777e763bac4bde67a48d1d40b5d15ba
Author: Indrajit Roychoudhury <iroychoudhury@...>
Date: 2018-07-11T03:16:45Z
Add tests, address review comments
----
---
