Hi, I have HDFS file of size 598MB. I create RDD over this file and cache it in RAM in a 7 node cluster with 2G RAM each. I find that each partition gets replicated thrice or even 4 times in the cluster even without me specifying in code. Total partitions are 5 for the RDD created but cached partitions are 17. I want to know if Spark replicates RDD automatically whenever there is scope (availability of extra resources)?
-- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Caching-RDD-tp21728.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
