Reynold Xin created SPARK-2321:
----------------------------------
Summary: Design a proper progress reporting & event listener API
Key: SPARK-2321
URL: https://issues.apache.org/jira/browse/SPARK-2321
Project: Spark
Issue Type: Improvement
Reporter: Reynold Xin
Priority: Critical
This is a ticket to track progress on redesigning the SparkListener and
JobProgressListener UI.
There are multiple problems with the current design, including:
0. I'm not sure if the API is usable in Java (there are at least some enums we
used in Scala and a bunch of case classes that might complicate things).
1. The whole API is marked as DeveloperApi, because we haven't paid a lot of
attention to it yet.
2. There is no easy way to connect jobs with stages. Similarly, there is no
easy way to connect job groups with jobs / stages.
3. JobProgressListener itself has no encapsulation at all. States can be
arbitrarily mutated by external programs. Variable names are sort of randomly
decided.
We should just revisit these and propose a new, concrete design.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
