Have you run the same test but with a URL in HDFS rather than the local filesystem? I think order may be preserved in that run, which makes the local filesystem losing order look more like a bug.
Sent from my mobile phone On Apr 25, 2014 9:11 AM, "Mingyu Kim" <[email protected]> wrote: > If the underlying file system returns files in a non-alphabetical order to > java.io.File.listFiles(), Spark reads the partitions out of order. Here’s > an example. > > var sc = new SparkContext(“local[3]”, “test”); > var rdd1 = sc.parallelize([1,2,3,4,5]); > rdd1.saveAsTextFile(“file://path/to/file”); > var rdd2 = sc.textFile(“file://path/to/file”); > rdd2.collect(); > > rdd1 is saved to file://path/to/file in three partitions. (I.e. > /path/to/file/part-00000, /path/to/file/part-00001, > /path/to/file/part-00002) Since File.listFiles(), which is used in > org.apache.hadoop.fs.RawLocalFileSystem.listStatus(), returns the > partitions out-of-order, rdd2 has the rows in the order different from that > of rdd1. Note that File.listFiles() explicitly says that it doesn’t > guarantee the order. > > The behavior of RawLocalFileSystem is fine for MapReduce jobs because they > don’t care about orders, but for Spark, which has a notion of row order, > this looks like a bug. The correct fix would be to sort the files after > calling File.listFiles(). > > This may be possible to fix somewhere by creating a wrapper > org.apache.hadoop.fs.FileSystem class that sorts the file list before > returning. Is this considered in the original design? Is this a bug? > > Mingyu >
