Kousuke Saruta created SPARK-31360:
--------------------------------------
Summary: Fix hung-up issue in StagePage
Key: SPARK-31360
URL: https://issues.apache.org/jira/browse/SPARK-31360
Project: Spark
Issue Type: Bug
Components: Web UI
Affects Versions: 3.1.0
Reporter: Kousuke Saruta
Assignee: Kousuke Saruta
WebUI will be hungup in the following situation.
1. Run a job with shuffle.
scala> sc.parallelize(1 to 10).map(x => (x, x)).reduceByKey(_ + _).collect
2. Visit StagePage for the stage writing shuffle data and check `Shuffle Write
Time`.
3. Run a job with no shuffle.
scala> sc.parallelize(1 to 10).collect
4. Visit StagePage for the last stage.
This issue is caused by following reason.
In stagepage.js, an array `optionalColumns` has indices for columns for
optional metrics.
If a stage doesn't perform shuffle read or write, the corresponding indices are
removed from the array.
StagePage doesn't try to create column for such metrics, even if the state of
corresponding optional metrics are preserved as "visible".
But, if a stage doesn't perform both shuffle read and write, the index for
`Shuffle Write Time` isn't removed.
In that case, StagePage tries to create a column for `Shuffle Write Time` even
though there are no metrics for shuffle write, leading hungup.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
