Re: DISCUSS: is the order in FS.listStatus() required to be sorted?

2015-06-16 Thread Colin McCabe
On Tue, Jun 16, 2015 at 3:02 AM, Steve Loughran ste...@hortonworks.com wrote: On 15 Jun 2015, at 21:22, Colin P. McCabe cmcc...@apache.org wrote: One possibility is that we could randomize the order of returned results in HDFS (at least within a given batch of results returned from the NN).

Re: DISCUSS: is the order in FS.listStatus() required to be sorted?

2015-06-16 Thread Steve Loughran
On 15 Jun 2015, at 21:22, Colin P. McCabe cmcc...@apache.org wrote: One possibility is that we could randomize the order of returned results in HDFS (at least within a given batch of results returned from the NN). This is similar to how the Go programming language randomizes the order of

Re: DISCUSS: is the order in FS.listStatus() required to be sorted?

2015-06-15 Thread Colin P. McCabe
On Mon, Jun 1, 2015 at 3:21 AM, Steve Loughran ste...@hortonworks.com wrote: HADOOP-12009 (https://issues.apache.org/jira/browse/HADOOP-12009) patches the FS javadoc and contract tests to say the order you get things back from a listStatus() isn't guaranteed to be alphanumerically sorted

Re: DISCUSS: is the order in FS.listStatus() required to be sorted?

2015-06-01 Thread Vinayakumar B
I think the patch just updates the doc as of now, not changing any code to affect the existing usage. Sorting depends on the underlying implementations. Linux *ls *implementation returns alphanumerically sorted array by default ( Current implementation might have assumed from here to sort by

DISCUSS: is the order in FS.listStatus() required to be sorted?

2015-06-01 Thread Steve Loughran
HADOOP-12009 (https://issues.apache.org/jira/browse/HADOOP-12009) patches the FS javadoc and contract tests to say the order you get things back from a listStatus() isn't guaranteed to be alphanumerically sorted That's one of those assumptions which we all have, but which, when you think

Re: DISCUSS: is the order in FS.listStatus() required to be sorted?

2015-06-01 Thread Allen Wittenauer
The POSIX spec for readdir (http://pubs.opengroup.org/onlinepubs/009695399/functions/readdir.html) doesn’t spell out a sort order, so it should be assumed that the ordering isn’t guaranteed. Chris Siebenmann has written a few relative blog posts on the topic that might be of