My spark streaming checkpoint directory is being written to HDFS with default replication factor of 3.
In my streaming application where I am listening from kafka and setting the dfs.replication = 2 as below the files are still being written with replication factor=3 SparkConf sparkConfig = new SparkConf().setMaster("mymaster").set("spark.hadoop.dfs.replication", "2"); Is there anything else that I need to do ?? Thanks !!! Abhi