HyukjinKwon commented on a change in pull request #28465:
URL: https://github.com/apache/spark/pull/28465#discussion_r421177883
##########
File path: docs/rdd-programming-guide.md
##########
@@ -360,7 +360,7 @@ Some notes on reading files with Spark:
* If using a path on the local filesystem, the file must also be accessible at
the same path on worker nodes. Either copy the file to all workers or use a
network-mounted shared file system.
-* All of Spark's file-based input methods, including `textFile`, support
running on directories, compressed files, and wildcards as well. For example,
you can use `textFile("/my/directory")`, `textFile("/my/directory/*.txt")`, and
`textFile("/my/directory/*.gz")`.
+* All of Spark's file-based input methods, including `textFile`, support
running on directories, compressed files, and wildcards as well. For example,
you can use `textFile("/my/directory")`, `textFile("/my/directory/*.txt")`, and
`textFile("/my/directory/*.gz")`. When multiple files are read, the order of
elements in the resulting RDD is not guaranteed, as files can be read in any
order. Within a partition, element order is respected.
Review comment:
Well, I think this isn't only the case for reading. The natural order
can only be preserved in some certain contexts. You can still keep the natural
order by setting a very high value to `spark.sql.files.openCostInBytes` and
`spark.sql.files.maxPartitionBytes`.
Spark does not guarantee its natural order in general. Actually, I think we
should have a separate section or page to publicly document this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
