rrusso2007 commented on a change in pull request #24672: [SPARK-27801][SQL]
Improve performance of InMemoryFileIndex.listLeafFiles for HDFS directories
with many files
URL: https://github.com/apache/spark/pull/24672#discussion_r288224365
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InMemoryFileIndex.scala
##########
@@ -274,7 +275,21 @@ object InMemoryFileIndex extends Logging {
// [SPARK-17599] Prevent InMemoryFileIndex from failing if path doesn't
exist
// Note that statuses only include FileStatus for the files and dirs
directly under path,
// and does not include anything else recursively.
- val statuses = try fs.listStatus(path) catch {
+ val statuses: Array[FileStatus] = try {
+ fs match {
+ // DistributedFileSystem overrides listLocatedStatus to make 1 single
call to namenode
+ // to retrieve the file status with the file block location. The
reason to still fallback
+ // to listStatus is because the default implementation would
potentially throw a
+ // FileNotFoundException which is better handled by doing the lookups
manually below.
+ case _: DistributedFileSystem =>
Review comment:
Before coming up with this change I was actually wondering why there was no
conf options to ignore locality and why it was required to do those extra
lookups. In situations where the spark job is running on a small set of servers
out of a big HDFS cluster, locality is a bit useless as the hit rate is super
low, the time to get block locations was getting out of control for us and
locality wait per task was slowing down the jobs.
I think this change to get the blocks from the namenode is good enough for
us now but I can definitely see the benefit of having a conf option to disable
locality entirely which would turn off fetching of getFileBlockLocations and
also use listStatus always instead of my optimization to use listLocatedStatus.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
