[
https://issues.apache.org/jira/browse/SPARK-35799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Venki Korukanti updated SPARK-35799:
------------------------------------
Description:
Metric {{allUpdatesTimeMs}} meant to capture the start to end walltime of the
operator {{FlatMapGroupsWithStateExec}}, but currently it just
[captures|https://github.com/apache/spark/blob/79362c4efcb6bd4b575438330a14a6191cca5e4b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FlatMapGroupsWithStateExec.scala#L121]
the iterator creation time.
Fix it to measure similar to how other stateful operators measure. Example one
[here|https://github.com/apache/spark/blob/79362c4efcb6bd4b575438330a14a6191cca5e4b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala#L406].
This measurement is not perfect due to the nature of the lazy iterator and
also includes the time the consumer operator spent in processing the current
operator output, but it should give a good signal when comparing the metric in
one microbatch to the metric in another microbatch.
was:
Metric `allUpdatesTimeMs` meant to capture the start to end walltime of the
operator `FlatMapGroupsWithStateExec`, but currently it just
[captures|https://github.com/apache/spark/blob/79362c4efcb6bd4b575438330a14a6191cca5e4b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FlatMapGroupsWithStateExec.scala#L121]
the iterator creation time.
Fix it to measure similar to how other stateful operators measure. Example one
[here|https://github.com/apache/spark/blob/79362c4efcb6bd4b575438330a14a6191cca5e4b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala#L406].
This measurement is not perfect due to the nature of the lazy iterator and
also includes the time the consumer operator spent in processing the current
operator output, but it should give a good signal when comparing the metric in
one microbatch to the metric in another microbatch.
> Fix the allUpdatesTimeMs metric measuring in FlatMapGroupsWithStateExec
> -----------------------------------------------------------------------
>
> Key: SPARK-35799
> URL: https://issues.apache.org/jira/browse/SPARK-35799
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 3.1.2
> Reporter: Venki Korukanti
> Priority: Minor
>
> Metric {{allUpdatesTimeMs}} meant to capture the start to end walltime of the
> operator {{FlatMapGroupsWithStateExec}}, but currently it just
> [captures|https://github.com/apache/spark/blob/79362c4efcb6bd4b575438330a14a6191cca5e4b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FlatMapGroupsWithStateExec.scala#L121]
> the iterator creation time.
> Fix it to measure similar to how other stateful operators measure. Example
> one
> [here|https://github.com/apache/spark/blob/79362c4efcb6bd4b575438330a14a6191cca5e4b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala#L406].
> This measurement is not perfect due to the nature of the lazy iterator and
> also includes the time the consumer operator spent in processing the current
> operator output, but it should give a good signal when comparing the metric
> in one microbatch to the metric in another microbatch.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
