Re: SparkListener onApplicationEnd processing an RDD throws exception because of stopped SparkContext

Ok, I understand. Yes, I will have to handle them in the main thread. Thanks! Sumona On Wed, Feb 17, 2016 at 12:24 PM Shixiong(Ryan) Zhu wrote: > `onApplicationEnd` is posted when SparkContext is stopping, and you cannot > submit any job to a stopping SparkContext.

Re: SparkListener onApplicationEnd processing an RDD throws exception because of stopped SparkContext

`onApplicationEnd` is posted when SparkContext is stopping, and you cannot submit any job to a stopping SparkContext. In general, SparkListener is used to monitor the job progress and collect job information, an you should not submit jobs there. Why not submit your jobs in the main thread? On

Re: SparkListener onApplicationEnd processing an RDD throws exception because of stopped SparkContext

Can anyone provide some insight into the flow of SparkListeners, specifically onApplicationEnd? I'm having issues with the SparkContext being stopped before my final processing can complete. Thanks! Sumona On Mon, Feb 15, 2016 at 8:59 AM Sumona Routh wrote: > Hi there, > I

SparkListener onApplicationEnd processing an RDD throws exception because of stopped SparkContext

Hi there, I am trying to implement a listener that performs as a post-processor which stores data about what was processed or erred. With this, I use an RDD that may or may not change during the course of the application. My thought was to use onApplicationEnd and then saveToCassandra call to