[
https://issues.apache.org/jira/browse/SPARK-56537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18077919#comment-18077919
]
Jungtaek Lim edited comment on SPARK-56537 at 5/3/26 8:58 PM:
--------------------------------------------------------------
[~pateldh6029]
I'd prefer the way the definition of metric can tell by itself - so 2.b seems
to be a way to go. Though we can see how more complication it will add and
revert back the decision if it's unnecessarily complicated.
For 5, I generally agree adding more parameters to copy() isn't great. Can we
probably add a new method in StateOperatorProgress to "copy" the instance but
zeroing out metrics for no execution trigger?
was (Author: kabhwan):
I'd prefer the way the definition of metric can tell by itself - so 2.b seems
to be a way to go. Though we can see how more complication it will add and
revert back the decision if it's unnecessarily complicated.
For 5, I generally agree adding more parameters to copy() isn't great. Can we
probably add a new method in StateOperatorProgress to "copy" the instance but
zeroing out metrics for no execution trigger?
> Some stateful operator metrics are not reset in no batch trigger progress
> event
> -------------------------------------------------------------------------------
>
> Key: SPARK-56537
> URL: https://issues.apache.org/jira/browse/SPARK-56537
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 4.2.0
> Reporter: Jungtaek Lim
> Priority: Minor
>
> [https://github.com/apache/spark/pull/55331#discussion_r3103933756]
> We reset several metrics (about num rows) on stateful operator metrics for no
> batch trigger progress event, but still we have more metrics to be audited.
> This ticket tracks the effort.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
