Ryan, Apologies for coming back so late, I created a github repo to resolve this problem. On trying your solution for making the pool a Singleton, I get a null pointer exception in the worker. Do you have any other suggestions, or a simpler mechanism for handling this?
I have put all the current code which was forked from an existing git repo here: https://github.com/nipunarora/Spark-Kafka-Writer There does seem to be duplicate creation of Kafka Writers in every micro-batch. Thanks Nipun P.S the version I shared before was writing JavaDStream<String>, the one in the github project writes JavaDStream<Tuple2<>> --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
