[
https://issues.apache.org/jira/browse/SPARK-26260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shahid updated SPARK-26260:
---------------------------
Summary: Summary Task Metrics for Stage Page: Efficient implementation for
SHS when using disk store. (was: Summary Task Metrics for Stage Page:
Efficient implimentation for SHS when using disk store.)
> Summary Task Metrics for Stage Page: Efficient implementation for SHS when
> using disk store.
> --------------------------------------------------------------------------------------------
>
> Key: SPARK-26260
> URL: https://issues.apache.org/jira/browse/SPARK-26260
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.4.0, 3.0.0
> Reporter: shahid
> Priority: Major
>
> Currently, tasks summary metrics is calculated based on all the tasks,
> instead of successful tasks.
> After the JIRA, https://issues.apache.org/jira/browse/SPARK-26119, when using
> InMemory store, it find task summary metrics for all the successful tasks
> metrics. But we need to find an efficient implementation for disk store case
> for SHS. The main bottle neck for disk store is deserialization time overhead.
> Hints: Need to rework on the way indexing works, so that we can index by
> specific metrics for successful and failed tasks differently (would be
> tricky). Also would require changing the disk store version (to invalidate
> old stores).
> OR any other efficient solutions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
