Chao Sun created SPARK-33135:
--------------------------------
Summary: Use listLocatedStatus from FileSystem implementations
Key: SPARK-33135
URL: https://issues.apache.org/jira/browse/SPARK-33135
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 3.0.1
Reporter: Chao Sun
{{HadoopFsUtils.parallelListLeafFiles}} currently only calls
{{listLocatedStatus}} when the {{FileSystem}} impl is {{DistributedFileSystem}}
or {{ViewFileSystem}}. For other types of {{FileSystem}}, it calls
{{listStatus}} and then subsequently calls {{getFileBlockLocations}} on all the
result {{FileStatus}}es.
In Hadoop client, {{listLocatedStatus}} is a well-defined API and in fact it is
often overridden by specific file system implementations, such as S3A. The
default {{listLocatedStatus}} also has similar behavior as it's done in Spark.
Therefore, instead of re-implement the logic in Spark itself, it's better to
rely on the {{FileSystem}}-specific implementation for {{listLocatedStatus}},
which could include its own optimizations in the code path.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
