[
https://issues.apache.org/jira/browse/FLINK-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16969349#comment-16969349
]
Yu Li commented on FLINK-12699:
-------------------------------
Side note: currently we are using {{MemorySegment.put/getInt/Long}} (which
follows the native byte order of the machine system) in {{SkipListUtils}}, it's
ok since w/o the proposed improvement here we won't copy files around during
restore, instead we directly ingest data read from DFS into heap memory and do
spill naturally. However, we should change to use
{{MemorySegment.put/getInt/LongLittleEndian}} explicitly to avoid the possible
byte order issue when copying the on-disk files around (considering the case of
heterogeneous cluster, when new TM to restore checkpoint may have different
native byte order from the old one)
> Reduce CPU consumption when snapshot/restore the spilled key-group
> ------------------------------------------------------------------
>
> Key: FLINK-12699
> URL: https://issues.apache.org/jira/browse/FLINK-12699
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / State Backends
> Reporter: Yu Li
> Assignee: PengFei Li
> Priority: Major
>
> We need to prevent the unnecessary de/serialization when
> snapshotting/restoring the spilled state key-group. To achieve this, we need
> to:
> 1. Add meta information for {{HeapKeyedStatebackend}} checkpoint on DFS,
> separating the on-heap and on-disk part
> 2. Write the off-heap bytes directly to DFS when checkpointing and mark it as
> on-disk
> 3. Directly write the bytes onto disk when restoring the data back from DFS,
> if it's marked as on-disk
> Notice that we cannot directly use file copy since we use mmap meanwhile
> support copy-on-write.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
