timvw opened a new issue, #3655: URL: https://github.com/apache/arrow-datafusion/issues/3655
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** Many object_stores are not as clean as expected. Eg: many have "hidden" files such as _SUCCESS or _temp/blahblah. When using ListingTableUrl these files are not filtered and cause issues when inferring schema/options. **Describe the solution you'd like** ListingTableUrl has an optional glob but it would be more helpful if this was somewhat more generic, eg an optional Fn(&ObjectMeta) -> bool **Describe alternatives you've considered** We could implement the filtering at other layers in the application but they all seem to come with drawbacks/feeling awkward (eg: such filtering does not really belong in a wrapper around an object_store). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
