[
https://issues.apache.org/jira/browse/FLINK-10473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17067767#comment-17067767
]
Andrey Zagrebin commented on FLINK-10473:
-----------------------------------------
Hi [~lsy]
I have attached link to the
[FLIP-25|[https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=80452561#FLIP-25:SupportUserStateTTLNatively-Incrementalcleanupwithglobaliterator(heapbackend)JiraFLINK-10473]]
with details about the feature design. The cleanup is applicable only for heap
state backend.
When I run TtlStateTestBase#testIncrementalCleanup for e.g.
HeapAsyncSnapshotTtlStateTest, debugger stops in
TtlIncrementalCleanup#runCleanup:81 on:
stateIterator.remove(state);
> State TTL incremental cleanup using Heap backend key iterator
> -------------------------------------------------------------
>
> Key: FLINK-10473
> URL: https://issues.apache.org/jira/browse/FLINK-10473
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / State Backends
> Affects Versions: 1.7.0
> Reporter: Andrey Zagrebin
> Assignee: Andrey Zagrebin
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.8.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This feature enables lazy background cleanup of state with time-to-live in
> state keyed backend which stores state in JVM heap. The idea is to keep a
> global state lazy iterator with loose consistency. Every time a state value
> for some key is accessed or a record is processed, the iterator is advanced,
> TTL of iterated state entries is checked and the expired entries are cleaned
> up. When the iterator reaches the end of state storage it just starts over.
> This way the state with TTL is regularly cleaned up to prevent ever growing
> memory consumption. The caveat of this cleanup strategy is that if state is
> not accessed or no records are processed then accumulated expired state still
> occupies the storage.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
