Siying Dong created SPARK-45084:
-----------------------------------
Summary: ProgressReport should include an accurate effective
shuffle partition number
Key: SPARK-45084
URL: https://issues.apache.org/jira/browse/SPARK-45084
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 3.4.2
Reporter: Siying Dong
Currently, there is a numShufflePartitions "metric" reported in
StateOperatorProgress part of the progress report. However, the number is
reported by aggregating executors so in the case of task retry or speculative
executor, the metric is higher than number of shuffle partitions for the query
plan. Number of shuffle partitions can be useful for reporting purpose so
having a metric is helpful.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
