Re: SparkStreaming onStart not being invoked on CustomReceiver attached to master with multiple workers

2015-04-20 Thread Tathagata Das
Responses inline. On Mon, Apr 20, 2015 at 3:27 PM, Ankit Patel patel7...@hotmail.com wrote: What you said is correct and I am expecting the printlns to be in my console or my SparkUI. I do not see it in either places. Can you actually login into the machine running the executor which runs the

RE: SparkStreaming onStart not being invoked on CustomReceiver attached to master with multiple workers

2015-04-20 Thread Ankit Patel
What you said is correct and I am expecting the printlns to be in my console or my SparkUI. I do not see it in either places. However, if you run the program then the printlns do print for the constructor of the receiver and the for the foreach statements with total count 0. When you run it in

RE: SparkStreaming onStart not being invoked on CustomReceiver attached to master with multiple workers

2015-04-20 Thread Ankit Patel
The code I've written is simple as it just invokes a thread and calls a store method on the Receiver class. I see this code with printlns working fine when I try spark-submit --jars jar --class test.TestCustomReceiver jar However it does not work with I try the same command above with --master

Re: SparkStreaming onStart not being invoked on CustomReceiver attached to master with multiple workers

2015-04-20 Thread Akhil Das
Would be good, if you can paste your custom receiver code and the code that you used to invoke it. Thanks Best Regards On Mon, Apr 20, 2015 at 9:43 AM, Ankit Patel patel7...@hotmail.com wrote: I am experiencing problem with SparkStreaming (Spark 1.2.0), the onStart method is never called on

SparkStreaming onStart not being invoked on CustomReceiver attached to master with multiple workers

2015-04-19 Thread Ankit Patel
I am experiencing problem with SparkStreaming (Spark 1.2.0), the onStart method is never called on CustomReceiver when calling spark-submit against a master node with multiple workers. However, SparkStreaming works fine with no master node set. Anyone notice this issue?