Roman Khachatryan created FLINK-23461:
-----------------------------------------
Summary: Consider disallowing in-memory state handles for
materialized state
Key: FLINK-23461
URL: https://issues.apache.org/jira/browse/FLINK-23461
Project: Flink
Issue Type: Sub-task
Components: Runtime / State Backends
Reporter: Roman Khachatryan
Fix For: 1.14.0
*For non-mateialized part*, FLINK-21353 doesn't use nor
FsCheckpointStreamFactory neither PlaceholderStreamStateHandle so it's not an
issue. Adding it in the future doesn't make sense as for such small changes
incremental checkpoint might work better.
*For materialized part, ByteStreamStateHandle* can be currently used. This can
bring back to life issues like FLINK-21351 - if checkpoint subsumption on TM
*will* be decoupled from the state backends state. Removing those assumptions
is one of the goals of changing the ownership.
An easy way to solve it is to just enforce zero threshold for writing to DFS
instead of memory.
*PlaceholderStreamStateHandle* can be used for the materialized state
(regardless of ByteStreamStateHandle; unless SnapshotStrategy is changed).
However, it shouldn't cause any issues:
- if the file is shared (i.e. after recovery) then by definition it should be
managed by JM
- otherwise, JM should still replace placeholders (FLINK-23137); and it should
have received the original state objects before; no re-upload should happen
(FLINK-23344) - so JM and TM will always refer to the same file
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
