Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/3009#issuecomment-63560812
Awesome -- sounds great!
On Tue, Nov 18, 2014 at 2:09 PM, Josh Rosen <[email protected]>
wrote:
> although there are possibly some race conditions here with the StageInfo
> being modified by the DAGScheduler concurrently with a listener using it
>
> I think this is actually okay: most of StageInfo's fields are immutable
> and the DAGScheduler already exposes its StageInfos to listeners through
> the SparkListenerStageSubmitted event. Passing the StageInfos in the
> onJobSubmitted event would let me fix another weird UI anomaly on the job
> details page where the UI knows about the existence of a stage but doesn't
> know its name / call site, etc. until that stage starts.
>
> This change (or one where we just store the number of tasks) cause some
> trickiness in the JSONProtocol backwards-compatibility support, but I
don't
> think that should be too tricky to address.
>
> â
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/3009#issuecomment-63554720>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
