[
https://issues.apache.org/jira/browse/FLINK-31245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17694018#comment-17694018
]
David Morávek commented on FLINK-31245:
---------------------------------------
Hi [~Zhanghao Chen] , thanks for the report!
As far as I'm aware, this was only used in the Kinesis source [1] for an early
version of watermark alignment. I'd favor deprecating it, but we need to run
this through the dev mailing list first in case there are other use cases I'm
unaware of.
[1]
https://github.com/apache/flink-connector-aws/blob/d0817fecdcaa53c4bf039761c2d1a16e8fb9f89b/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/util/JobManagerWatermarkTracker.java
> Adaptive scheduler does not reset the state of GlobalAggregateManager when
> rescaling
> ------------------------------------------------------------------------------------
>
> Key: FLINK-31245
> URL: https://issues.apache.org/jira/browse/FLINK-31245
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
> Affects Versions: 1.16.1
> Reporter: Zhanghao Chen
> Priority: Major
> Fix For: 1.18.0
>
>
> *Problem*
> GlobalAggregateManager is used to share state amongst parallel tasks in a job
> and thus coordinate their execution. It maintains a state (the _accumulators_
> field in JobMaster) in JM memory. The accumulator state content is defined in
> user code, in my company, a user stores task parallelism in the accumulator,
> assuming task parallelism never changes. However, this assumption is broken
> when using adaptive scheduler.
> *Possible Solutions*
> # Mark GlobalAggregateManager as deprecated. It seems that operator
> coordinator can completely replace GlobalAggregateManager and is a more
> elegent solution. Therefore, it is fine to deprecate GlobalAggregateManager
> and leave this issue there. If that's the case, we can open another ticket
> for doing that.
> # If we decide to continue supporting GlobalAggregateManager, then we need
> to reset the state when rescaling.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
