[
https://issues.apache.org/jira/browse/FLINK-24815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17447537#comment-17447537
]
Roman Khachatryan commented on FLINK-24815:
-------------------------------------------
I agree think that passing some magic number instead of precomputed one in case
of rescaling is error-prone. And moving computation to the getter (and caching
the results) sounds safer but CPU spike during rendering can be misleading.
And it probably won't help if getStateSize() would be called anyways, for
example to add the checkpoint to the history?
Another direction is to optimize
IncrementalRemoteKeyedStateHandle.getStateSize(). IIUC, the problem is
particularly with this handle.
[~Ming Li] could you please confirm it (maybe by flamegraph or profiler)?
For example, we could store state size in metadata (and compute once on TM). As
long as getIntersection() returns the same object (it does currently), this
should be enough.
> Reduce the cpu cost of calculating stateSize during state allocation
> --------------------------------------------------------------------
>
> Key: FLINK-24815
> URL: https://issues.apache.org/jira/browse/FLINK-24815
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing, Runtime / State Backends
> Affects Versions: 1.14.0
> Reporter: ming li
> Priority: Major
>
> When the task failover, we will reassign the state for each subtask and
> create a new {{OperatorSubtaskState}} object. At this time, the {{stateSize}}
> field in the {{OperatorSubtaskState}} will be recalculated. When using
> incremental {{{}Checkpoint{}}}, this field needs to traverse all shared
> states and then accumulate the size of the state.
> Taking a job with 2000 parallelism and 100 share state for each task as an
> example, it needs to traverse 2000 * 100 = 20w times. At this time, the cpu
> of the JM scheduling thread will be full.
> I think we can try to provide a construction method with {{stateSize}} for
> {{OperatorSubtaskState}} or delay the calculation of {{{}stateSize{}}}.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
