sunchao commented on pull request #29471: URL: https://github.com/apache/spark/pull/29471#issuecomment-677961496
Thanks @viirya - fixed the PR description. @holdenk this is ready for review now. I removed the changes on `WholeTextFileInputFormat` to make this focusing on refactoring (i.e., moving the code from SQL to core). We can do follow-ups if later on we decide to swap the listing logic for Hadoop file input formats as well. Other than that, a few fixes from the version I inherited from you: - fixed a NPE when `BlockLocations` is null - I think we should not convert `FileStatus` to `LocatedFileStatus` and initialize block locations with null. - changed `ignoreDataLocality` default to false, to match that in SQL - removed extra handling of `FileNotFoundException` which seems not necessary since the default behavior of `FileSystem` won't throw the exception, as mentioned offline. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
