[
https://issues.apache.org/jira/browse/SPARK-2321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reynold Xin updated SPARK-2321:
-------------------------------
Affects Version/s: 1.0.0
> Design a proper progress reporting & event listener API
> -------------------------------------------------------
>
> Key: SPARK-2321
> URL: https://issues.apache.org/jira/browse/SPARK-2321
> Project: Spark
> Issue Type: Improvement
> Affects Versions: 1.0.0
> Reporter: Reynold Xin
> Priority: Critical
>
> This is a ticket to track progress on redesigning the SparkListener and
> JobProgressListener API.
> There are multiple problems with the current design, including:
> 0. I'm not sure if the API is usable in Java (there are at least some enums
> we used in Scala and a bunch of case classes that might complicate things).
> 1. The whole API is marked as DeveloperApi, because we haven't paid a lot of
> attention to it yet. Something as important as progress reporting deserves a
> more stable API.
> 2. There is no easy way to connect jobs with stages. Similarly, there is no
> easy way to connect job groups with jobs / stages.
> 3. JobProgressListener itself has no encapsulation at all. States can be
> arbitrarily mutated by external programs. Variable names are sort of randomly
> decided and inconsistent.
> We should just revisit these and propose a new, concrete design.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
