tustvold commented on issue #3261: URL: https://github.com/apache/arrow-datafusion/issues/3261#issuecomment-1227360620
The reason I didn't do this is glob characters aren't URL-safe, so something like `s3://bucket/path/*.parquet` isn't a valid URL. I could only find examples of systems that supported glob expressions to local filesystem, and so I wasn't really sure how best to encode globs in URLs and opted to just punt on it. Some possible ideas: * Just ignore that it isn't a valid URL and accept the fact it is potentially very confusing (what this ticket proposes) * Provide a programmatic interface to construct a `ListingTableUrl` with a custom scheme and glob * Encode the glob expression as a URL-encoded query parameter * Something else It is also potentially worth highlighting that IIRC the logical plan serialization currently doesn't handle glob expressions and just drops them on the floor. I think it would really help move this forward if we could find an example of a system that supports glob expressions to object stores, otherwise we end up having to design something custom which we will inevitably get wrong -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
