steveloughran commented on pull request #29471: URL: https://github.com/apache/spark/pull/29471#issuecomment-684831484
BTW, wrote something up on listing. https://github.com/steveloughran/engineering-proposals/blob/trunk/listing-performance.md anywhere you do listStatus(path): List[FileStatus], switch to listStatusIterator, but, if the returned iterator is Closeable, make sure you close it after. Then I or a someone else will not only add the s3a and abfs speedups (alongside today's HDFS), I'll do the same for the local FS. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
