[
https://issues.apache.org/jira/browse/FLINK-10132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16578367#comment-16578367
]
ASF GitHub Bot commented on FLINK-10132:
----------------------------------------
azagrebin opened a new pull request #6548: [FLINK-10132] Incremental cleanup of
local expired state with TTL dis…
URL: https://github.com/apache/flink/pull/6548
## What is the purpose of the change
This PR extends the cleanup of full snapshot from the expired state. A queue
is added between a snapshotting thread which discovers expired state during its
full scan and main thread which accesses state. The main thread incrementally
pulls the queue and cleans up local state if still applicable.
## Brief change log
- add FullSnapshotCleanupStrategy config for this feature, users can
activate it in state config builder
StateTtlConfig.Builder.cleanupFullSnapshotAndLocalState
- Add key and namespace parameters to StateSnapshotTransformer interface
to save expired keys in queue and address state later for cleanup
- Parse key/namespace bytes in RocksDb and pass them in both backends to
StateSnapshotTransformer
- add TtlIncrementalCleanup service which wraps queue with expiration
specific add/pull logic
- add call from TtlStateSnapshotTransformer to
TtlIncrementalCleanup.addIfExpired to enqueue expired keys
- add callback from states to TtlIncrementalCleanup.stateAccessed when
state is touched
- add check/cleanup in TTL state wrappers when pulled from queue
AbstractTtlState.cleanupIfExpired
- add test for the feature TtlStateTestBase.testFullSnapshotCleanupQueue
## Verifying this change
unit tests of TTL logic.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (yes)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (yes)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (yes)
- If yes, how is the feature documented? (not documented)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Incremental cleanup of local expired state with TTL discovered in full
> snapshot
> --------------------------------------------------------------------------------
>
> Key: FLINK-10132
> URL: https://issues.apache.org/jira/browse/FLINK-10132
> Project: Flink
> Issue Type: Improvement
> Components: State Backends, Checkpointing
> Affects Versions: 1.7.0
> Reporter: Andrey Zagrebin
> Assignee: Andrey Zagrebin
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> This issue suggests to extend approach from FLINK-9938 and add a queue
> between a snapshotting thread which discovers expired state during its full
> scan and main thread which accesses state and can incrementally pull the
> queue and clean up local state if still applicable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
