Have you run the same test but with a URL in HDFS rather than the local
filesystem? I think order may be preserved in that run, which makes the
local filesystem losing order look more like a bug.

Sent from my mobile phone
On Apr 25, 2014 9:11 AM, "Mingyu Kim" <[email protected]> wrote:

> If the underlying file system returns files in a non-alphabetical order to
> java.io.File.listFiles(), Spark reads the partitions out of order. Here’s
> an example.
>
> var sc = new SparkContext(“local[3]”, “test”);
> var rdd1 = sc.parallelize([1,2,3,4,5]);
> rdd1.saveAsTextFile(“file://path/to/file”);
> var rdd2 = sc.textFile(“file://path/to/file”);
> rdd2.collect();
>
> rdd1 is saved to file://path/to/file in three partitions. (I.e.
> /path/to/file/part-00000, /path/to/file/part-00001,
>  /path/to/file/part-00002) Since File.listFiles(), which is used in
> org.apache.hadoop.fs.RawLocalFileSystem.listStatus(), returns the
> partitions out-of-order, rdd2 has the rows in the order different from that
> of rdd1. Note that File.listFiles() explicitly says that it doesn’t
> guarantee the order.
>
> The behavior of RawLocalFileSystem is fine for MapReduce jobs because they
> don’t care about orders, but for Spark, which has a notion of row order,
> this looks like a bug. The correct fix would be to sort the files after
> calling File.listFiles().
>
> This may be possible to fix somewhere by creating a wrapper
> org.apache.hadoop.fs.FileSystem class that sorts the file list before
> returning. Is this considered in the original design? Is this a bug?
>
> Mingyu
>

Reply via email to