Hi Elias,
let me answer the questions to the best of my knowledge, but in general I
think this is as expected.
(Let me give a link to the docs explaining the activation [1] for other
readers first.)
On Friday, 3 November 2017 01:11:52 CET Elias Levy wrote:
> What is the interaction of incremental checkpointing and external
> checkpoints?
Externalized checkpoints may be incremental [2] (I'll fix the formatting error
that is not rendering the arguments as a list, making them less visible)
> Any interaction with the state.checkpoints.num-retained config?
Yes, this remains the number of available checkpoints. There may, however, be
more folders containing RocksDB state that was originally put into checkpoint
X but is also still required in checkpoint X+10 or so. These files will be
cleaned up once they are not needed anymore.
> Does incremental checkpointing require any maintenance?
No, state is cleaned up once it is not used/referenced anymore.
> Any interaction with savepoints?
No, a savepoint uses Flink's own data format and is not incremental [3].
> Does it perform better against certain "file systems"? E.g. it S3 not
> recommended for it? How about EFS?
I can't think of a reason this should be any different to non-incremental
checkpoints. Maybe Stefan (cc'd) has some more info on this.
For more details on the whole topic, I can recommend Stefan's talk at the last
Flink Forward [4] though.
Nico
[1] https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/state/
large_state_tuning.html#tuning-rocksdb
[2] https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/state/
checkpoints.html#difference-to-savepoints
[3] https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/state/
savepoints.html
[4] https://www.youtube.com/watch?
v=dWQ24wERItM=36=PLDX4T_cnKjD0JeULl1X6iTn7VIkDeYX_X
signature.asc
Description: This is a digitally signed message part.
