Thanks for the reply. It indeed increased the usage. There was another issue we found, we were broadcasting hadoop configuration by writing a wrapper class over it. But found the proper way in Spark Code
sc.broadcast(new SerializableWritable(conf)) -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-running-slow-for-small-hadoop-files-of-10-mb-size-tp4526p4811.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
