Re: Graceful shutdown drops processing in Spark Streaming

Thanks! Done. https://issues.apache.org/jira/browse/SPARK-10995 On 7 October 2015 at 21:24, Tathagata Das wrote: > Aaah, interesting, you are doing 15 minute slide duration. Yeah, > internally the streaming scheduler waits for the last "batch" interval > which has data to

Re: Graceful shutdown drops processing in Spark Streaming

Aaah, interesting, you are doing 15 minute slide duration. Yeah, internally the streaming scheduler waits for the last "batch" interval which has data to be processed, but if there is a sliding interval (i.e. 15 mins) that is higher than batch interval, then that might not be run. This is indeed a

Graceful shutdown drops processing in Spark Streaming

After triggering the graceful shutdown on the following application, the application stops before the windowed stream reaches its slide duration. As a result, the data is not completely processed (i.e. saveToMyStorage is not called) before shutdown. According to the documentation, graceful