[
https://issues.apache.org/jira/browse/SPARK-7951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14565475#comment-14565475
]
Josh Rosen commented on SPARK-7951:
-----------------------------------
This sounds like a possible duplicate of SPARK-5190, which has partially been
mitigated by my patch for SPARK-5411.
> Events Missing with custom sparkListener
> ----------------------------------------
>
> Key: SPARK-7951
> URL: https://issues.apache.org/jira/browse/SPARK-7951
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.3.0
> Reporter: Anubhav Srivastava
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> If we try to add our own SparkListener, we miss some of the events. I went
> through the code for spark-core, figured out that initial events like
> onApplicationStart() etc are fired when SparkContext() is initialised.
> Default eventListeners are getting those events as they are added while the
> sparkContext object is created, but when we add our own listener by calling
> sc().addListener(), while firing initial events our listener is not present
> at that time so its not getting those events.
> In some cases you may be able to get those events as when asyncListenerBus
> is created, it spawns a new thread so in case the execution of that thread
> takes time, our listener might get added by that time so we may receive those
> events but that's not fair.
> I Can fix it. In-fact the fix is working fine for me. But just wanted to know
> is that done purposely?? If yes then what's the actual reason???
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
