[
https://issues.apache.org/jira/browse/SPARK-31360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated SPARK-31360:
-----------------------------------
Description:
StagePage will be hung-up with following operations.
1. Run a job with shuffle.
scala> sc.parallelize(1 to 10).map(x => (x, x)).reduceByKey(_ + _).collect
2. Visit StagePage for the stage writing shuffle data and check `Shuffle Write
Time`.
3. Run a job with no shuffle.
scala> sc.parallelize(1 to 10).collect
4. Visit StagePage for the last stage.
This issue is caused by following reason.
In stagepage.js, an array `optionalColumns` has indices for columns for
optional metrics.
If a stage doesn't perform shuffle read or write, the corresponding indices are
removed from the array.
StagePage doesn't try to create column for such metrics, even if the state of
corresponding optional metrics are preserved as "visible".
But, if a stage doesn't perform both shuffle read and write, the index for
`Shuffle Write Time` isn't removed.
In that case, StagePage tries to create a column for `Shuffle Write Time` even
though there are no metrics for shuffle write, leading hungup.
was:
WebUI will be hungup in the following situation.
1. Run a job with shuffle.
scala> sc.parallelize(1 to 10).map(x => (x, x)).reduceByKey(_ + _).collect
2. Visit StagePage for the stage writing shuffle data and check `Shuffle Write
Time`.
3. Run a job with no shuffle.
scala> sc.parallelize(1 to 10).collect
4. Visit StagePage for the last stage.
This issue is caused by following reason.
In stagepage.js, an array `optionalColumns` has indices for columns for
optional metrics.
If a stage doesn't perform shuffle read or write, the corresponding indices are
removed from the array.
StagePage doesn't try to create column for such metrics, even if the state of
corresponding optional metrics are preserved as "visible".
But, if a stage doesn't perform both shuffle read and write, the index for
`Shuffle Write Time` isn't removed.
In that case, StagePage tries to create a column for `Shuffle Write Time` even
though there are no metrics for shuffle write, leading hungup.
> Fix hung-up issue in StagePage
> ------------------------------
>
> Key: SPARK-31360
> URL: https://issues.apache.org/jira/browse/SPARK-31360
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 3.1.0
> Reporter: Kousuke Saruta
> Assignee: Kousuke Saruta
> Priority: Major
>
> StagePage will be hung-up with following operations.
> 1. Run a job with shuffle.
> scala> sc.parallelize(1 to 10).map(x => (x, x)).reduceByKey(_ + _).collect
> 2. Visit StagePage for the stage writing shuffle data and check `Shuffle
> Write Time`.
> 3. Run a job with no shuffle.
> scala> sc.parallelize(1 to 10).collect
> 4. Visit StagePage for the last stage.
> This issue is caused by following reason.
> In stagepage.js, an array `optionalColumns` has indices for columns for
> optional metrics.
> If a stage doesn't perform shuffle read or write, the corresponding indices
> are removed from the array.
> StagePage doesn't try to create column for such metrics, even if the state of
> corresponding optional metrics are preserved as "visible".
> But, if a stage doesn't perform both shuffle read and write, the index for
> `Shuffle Write Time` isn't removed.
> In that case, StagePage tries to create a column for `Shuffle Write Time`
> even though there are no metrics for shuffle write, leading hungup.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
