steveloughran commented on pull request #29959: URL: https://github.com/apache/spark/pull/29959#issuecomment-707163511
One thing to know here is that while listStatus etc is slow against object stores, it's not enough of a bandwidth killer than you get much speedup executing it across a spark cluster. Same process-many-threads: yes. Remote exec: no. I am going to make this class: org.apache.hadoop.mapred.LocatedFileStatusFetcher @Public/Evolving: in this PR: https://github.com/apache/hadoop/pull/2324 it does do multithread scanning today and is fairly stable. It's not the perfect API -I'd have a builder and actually return a remoteIterator of statuses which would block while new listing entries arrived, but it's there. With that PR will also collect aggregate stats on IO operations from any FS whose list calls provide them. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
