Siying Dong created SPARK-45084: ----------------------------------- Summary: ProgressReport should include an accurate effective shuffle partition number Key: SPARK-45084 URL: https://issues.apache.org/jira/browse/SPARK-45084 Project: Spark Issue Type: Improvement Components: Structured Streaming Affects Versions: 3.4.2 Reporter: Siying Dong
Currently, there is a numShufflePartitions "metric" reported inĀ StateOperatorProgress part of the progress report. However, the number is reported by aggregating executors so in the case of task retry or speculative executor, the metric is higher than number of shuffle partitions for the query plan. Number of shuffle partitions can be useful for reporting purpose so having a metric is helpful. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org