The error is resolved. I was using a comparator which was not serialised because of which it was throwing the error.
I have now switched to kryo serializer as it is faster than java serialser. I have set the required config conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer"); conf.set("spark.kryo.registrator", "MyRegistrator"); and also in MyRegistrator class I have registered all the classes I am serialising. How can I confirm that my code is actually using kryo serialiser and not java serialiser now ? PS : It seems like my code is still not using kryo serialiser. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Error-related-to-serialisation-in-spark-streaming-tp6801p6904.html Sent from the Apache Spark User List mailing list archive at Nabble.com.