Hi, I’m running spark on cluster mode containing 4 nodes, and trying to write CSV files to node’s local path (not HDFS). I’m spark.write.csv to write CSV files.
On master node: spark.write.csv creates a folder with csv file name and writes many files with part-r-000n suffix. This is okay for me, I can merge them later. But on worker nodes: spark.write.csv creates a folder with csv file name and writes many folders and files under _temporary/0/. This is not okay for me. Could someone please suggest me what could have been going wrong in my settings/how to be able to write csv files to the specified folder, and not to subfolders (_temporary/0/task_xxx) in worker machines. Thank you, Hemanth