HI All,
I have a spark streaming job, which is running fine. And in that job I am
using FAIR scheduler with multiple pools. One pool for main streaming job.
And another for some parallel batch processing . All is working as
expected, except ..
"The custom receiver that I used is not assigned to the pool used for
streaming."
I am using stream.foreachRDD.. { rdd =>
rdd.sparkContext.setLocalProperty("spark.scheduler.pool","streaming")
}
What I can understand is , I am setting the Thread local after Spark Custom
Receiver is initialized.
Receiver is running on default pool.
Setting Receiver using : ssc.receiverStream(MyCustomReceiver())
Any Help ?
Regards,
Rabin Banerjee
