[
https://issues.apache.org/jira/browse/FLINK-8531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephan Ewen resolved FLINK-8531.
---------------------------------
Resolution: Fixed
Fixed as of 4e481a72c1ed3cc5f177b511e5a72cd8726cf976
Consists of steps
- 99495c91ecce7141ae8b2fbc96492681a9d130bd
- 35c7d93ee85aa8689e804b713affa65b46af1acc
- 9903c8c42793b922549835217c586c5928999ea5
- 5cc50934bdcf80ae1fa69abe69e2f214852653f9
- bb19e7f5278d43cd4fd265e3d2afa2fcc793ccf5
- 1887187f6b5c210d2091c69ef14fa8b8a5cae82c
- fc21423e1f8f1a1661badef20f9c6f368f6daf8b
- e0b0f45bd9c8b06bd2cda56f6859d0d3944aa00e
- 4e481a72c1ed3cc5f177b511e5a72cd8726cf976
> Support separation of "Exclusive", "Shared" and "Task owned" state
> ------------------------------------------------------------------
>
> Key: FLINK-8531
> URL: https://issues.apache.org/jira/browse/FLINK-8531
> Project: Flink
> Issue Type: Sub-task
> Components: State Backends, Checkpointing
> Reporter: Stephan Ewen
> Assignee: Stephan Ewen
> Priority: Major
> Fix For: 1.5.0
>
>
> Currently, all state created at a certain checkpoint goes into the directory
> {{chk-id}}.
> With incremental checkpointing, some state is shared across checkpoint and is
> referenced by newer checkpoints. That way, old {{chk-id}} directories stay
> around, containing some shared chunks. That makes it both for users and
> cleanup hooks hard to determine when a {{chk-x}} directory could be deleted.
> The same holds for state that can only every be dropped by certain operators
> on the TaskManager, never by the JobManager / CheckpointCoordinator. Examples
> of that state are write ahead logs, which need to be retained until the move
> to the target system is complete, which may in some cases be later then when
> the checkpoint that created them is disposed.
> I propose to introduce different scopes for tasks:
> - **EXCLUSIVE** is for state that belongs to one checkpoint only
> - **SHARED** is for state that is possibly part of multiple checkpoints
> - **TASKOWNED** is for state that must never by dropped by the JobManager.
> For file based checkpoint targets, I propose that we have the following
> directory layout:
> {code}
> /user-defined-checkpoint-dir
> |
> + --shared/
> + --taskowned/
> + --chk-00001/
> + --chk-00002/
> + --chk-00003/
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
