dispanser commented on issue #1384: URL: https://github.com/apache/arrow-datafusion/issues/1384#issuecomment-1059759211
I have a project with similar requirements, where files / partitions are determined by looking up a catalog with additional indexing capabilities to determine the files to read prior to passing them to datafusion. In our case, we don't actually want to register a table (as the selection of files changes with every query), but need something like `read_parquet(Vec<Path>)`. I'm not sure if that makes it sufficiently different to what's discussed here to make it a separate issue, but it sounds to require very similar functionality under the hood. I'm happy to help, but not well-versed in the datafusion codebase to drive the design decisions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
