cchighman commented on pull request #28841: URL: https://github.com/apache/spark/pull/28841#issuecomment-695158666
> > I haven't looked into the detail yet, but Hadoop FS and a PathFilter API, does Spark file source support the path filter option? > > Hadoop's PathFilter only gives the user Path without metadata. But this is an interesting point. The user needs a file metadata filter essentially. For example, some user may ask to filter files using the file size. But supporting a filter in multiple languages is hard. @zsxwing Circling back around, you introduced the _latestFirst_option for determining how a file-based data source would process files. Any thoughts surrounding this feature from your end ? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
