[Spark Streaming] Are SparkListener/StreamingListener callbacks called concurrently?

[Majid Azimi]

Hi,

What is the concurrency model behind SparkListener or StreamingListener 
callbacks?
1. Multiple threads might access callbacks simultaneously.
2. Callbacks are guaranteed to be executed by a single thread.(Thread ids might 
change on consecutive calls, though)

I asked the same question on stackoverflow, and waited for about one day. Since 
there was no response, I'm reposting it here.

https://stackoverflow.com/questions/50921585/are-sparklistener-streaminglistener-callbacks-called-concurrently

Re: StreamingListener, anyone?

[Akhil Das]

Hi

Here's a working example: https://gist.github.com/akhld/b10dc491aad1a2007183

[image: Inline image 1]

Thanks
Best Regards

On Wed, Jun 3, 2015 at 10:09 PM, dgoldenberg dgoldenberg...@gmail.com
wrote:

 Hi,

 I've got a Spark Streaming driver job implemented and in it, I register a
 streaming listener, like so:

 JavaStreamingContext jssc = new JavaStreamingContext(sparkConf,
Durations.milliseconds(params.getBatchDurationMillis()));
 jssc.addStreamingListener(new JobListener(jssc));

 where JobListener is defined like so
 private static class JobListener implements StreamingListener {

 private JavaStreamingContext jssc;

 JobListener(JavaStreamingContext jssc) {
 this.jssc = jssc;
 }

 @Override
 public void
 onBatchCompleted(StreamingListenerBatchCompleted
 batchCompleted) {
 System.out.println( Batch completed.);
 jssc.stop(true);
 System.out.println( The job has been stopped.);
 }
 

 I do not seem to be seeing onBatchCompleted being triggered.  Am I doing
 something wrong?

 In this particular case, I was trying to implement a bulk ingest type of
 logic where the first batch is all we're interested in (reading out of a
 Kafka topic with offset reset set to smallest).




 --
 View this message in context:
 http://apache-spark-user-list.1001560.n3.nabble.com/StreamingListener-anyone-tp23140.html
 Sent from the Apache Spark User List mailing list archive at Nabble.com.

 -
 To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
 For additional commands, e-mail: user-h...@spark.apache.org

Re: StreamingListener, anyone?

[Shixiong Zhu]

You should not call `jssc.stop(true);` in a StreamingListener. It will
cause a dead-lock: `jssc.stop` won't return until `listenerBus` exits. But
since `jssc.stop` blocks `StreamingListener`, `listenerBus` cannot exit.

Best Regards,
Shixiong Zhu

2015-06-04 0:39 GMT+08:00 dgoldenberg dgoldenberg...@gmail.com:

 Hi,

 I've got a Spark Streaming driver job implemented and in it, I register a
 streaming listener, like so:

 JavaStreamingContext jssc = new JavaStreamingContext(sparkConf,
Durations.milliseconds(params.getBatchDurationMillis()));
 jssc.addStreamingListener(new JobListener(jssc));

 where JobListener is defined like so
 private static class JobListener implements StreamingListener {

 private JavaStreamingContext jssc;

 JobListener(JavaStreamingContext jssc) {
 this.jssc = jssc;
 }

 @Override
 public void
 onBatchCompleted(StreamingListenerBatchCompleted
 batchCompleted) {
 System.out.println( Batch completed.);
 jssc.stop(true);
 System.out.println( The job has been stopped.);
 }
 

 I do not seem to be seeing onBatchCompleted being triggered.  Am I doing
 something wrong?

 In this particular case, I was trying to implement a bulk ingest type of
 logic where the first batch is all we're interested in (reading out of a
 Kafka topic with offset reset set to smallest).




 --
 View this message in context:
 http://apache-spark-user-list.1001560.n3.nabble.com/StreamingListener-anyone-tp23140.html
 Sent from the Apache Spark User List mailing list archive at Nabble.com.

 -
 To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
 For additional commands, e-mail: user-h...@spark.apache.org

Re: StreamingListener, anyone?

[Dmitry Goldenberg]

Shixiong,

Thanks, interesting point. So if we want to only process one batch then
terminate the consumer, what's the best way to achieve that? Presumably the
listener could set a flag on the driver notifying it that it can terminate.
But the driver is not in a loop, it's basically blocked in
awaitTermination.  So what would be a way to trigger the termination in the
driver?

context.awaitTermination() allows the current thread to wait for the
termination of a context by stop() or by an exception - presumably, we
need to call stop() somewhere or perhaps throw.

Cheers,
- Dmitry

On Thu, Jun 4, 2015 at 3:55 AM, Shixiong Zhu zsxw...@gmail.com wrote:

 You should not call `jssc.stop(true);` in a StreamingListener. It will
 cause a dead-lock: `jssc.stop` won't return until `listenerBus` exits. But
 since `jssc.stop` blocks `StreamingListener`, `listenerBus` cannot exit.

 Best Regards,
 Shixiong Zhu

 2015-06-04 0:39 GMT+08:00 dgoldenberg dgoldenberg...@gmail.com:

 Hi,

 I've got a Spark Streaming driver job implemented and in it, I register a
 streaming listener, like so:

 JavaStreamingContext jssc = new JavaStreamingContext(sparkConf,
Durations.milliseconds(params.getBatchDurationMillis()));
 jssc.addStreamingListener(new JobListener(jssc));

 where JobListener is defined like so
 private static class JobListener implements StreamingListener {

 private JavaStreamingContext jssc;

 JobListener(JavaStreamingContext jssc) {
 this.jssc = jssc;
 }

 @Override
 public void
 onBatchCompleted(StreamingListenerBatchCompleted
 batchCompleted) {
 System.out.println( Batch completed.);
 jssc.stop(true);
 System.out.println( The job has been
 stopped.);
 }
 

 I do not seem to be seeing onBatchCompleted being triggered.  Am I doing
 something wrong?

 In this particular case, I was trying to implement a bulk ingest type of
 logic where the first batch is all we're interested in (reading out of a
 Kafka topic with offset reset set to smallest).




 --
 View this message in context:
 http://apache-spark-user-list.1001560.n3.nabble.com/StreamingListener-anyone-tp23140.html
 Sent from the Apache Spark User List mailing list archive at Nabble.com.

 -
 To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
 For additional commands, e-mail: user-h...@spark.apache.org

StreamingListener, anyone?

[dgoldenberg]

Hi,

I've got a Spark Streaming driver job implemented and in it, I register a
streaming listener, like so:

JavaStreamingContext jssc = new JavaStreamingContext(sparkConf,
   Durations.milliseconds(params.getBatchDurationMillis()));
jssc.addStreamingListener(new JobListener(jssc));

where JobListener is defined like so
private static class JobListener implements StreamingListener {

private JavaStreamingContext jssc;

JobListener(JavaStreamingContext jssc) {
this.jssc = jssc;
}

@Override
public void onBatchCompleted(StreamingListenerBatchCompleted
batchCompleted) {
System.out.println( Batch completed.);
jssc.stop(true);
System.out.println( The job has been stopped.);
}


I do not seem to be seeing onBatchCompleted being triggered.  Am I doing
something wrong?

In this particular case, I was trying to implement a bulk ingest type of
logic where the first batch is all we're interested in (reading out of a
Kafka topic with offset reset set to smallest).




--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/StreamingListener-anyone-tp23140.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: StreamingListener

[Akhil Das]

At the end of foreachrdd i believe.

Thanks
Best Regards

On Thu, Mar 12, 2015 at 6:48 AM, Corey Nolet cjno...@gmail.com wrote:

 Given the following scenario:

 dstream.map(...).filter(...).window(...).foreachrdd()

 When would the onBatchCompleted fire?

StreamingListener

[Corey Nolet]

Given the following scenario:

dstream.map(...).filter(...).window(...).foreachrdd()

When would the onBatchCompleted fire?

Example usage of StreamingListener

[Hafiz Mujadid]

Hi!

does anybody has some useful example of StreamingListener interface. When
and how can we use this interface to stop streaming when one batch of data
is processed?

Thanks alot



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Example-usage-of-StreamingListener-tp20357.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: Example usage of StreamingListener

[Hafiz Mujadid]

Thanks Akhil  
You are so helping Dear. 




--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Example-usage-of-StreamingListener-tp20357p20362.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org