pepijnve commented on PR #27210: URL: https://github.com/apache/datafusion/pull/27210#issuecomment-6793973613
> Would this be worth exploring before committing to the version-specific APIs? I’m motivated to help make it work, and have outlined related query-scoped catalog/provider limitations I'll see what I can do. Certainly doesn't hurt to have a think about how we could generalise the concept introduced here. One benefit to the approach taken in the current shape of the PR is that the design is very straightforward to implement and use. Assuming you're not replacing any core components, only the `SchemaProvider` needs to actively participate. It would be nice if we could retain that ease-of-use. I'm going to try parroting back your suggestion to make sure I understand it correctly. Could you confirm if I got the gist of it right? - Rather than going version/timestamp specific, we introduce a more general notion of a more specific request for a table (`TableResolutionRequest`) - To keep this extensible this will have to be an extensible type. Could be an enum with an extension variant a la `LogicalPlan::Extension` or alternatively it could be a trait object. - The conversion from SQL AST (`TableFactor`) to the resolution request is also an extension point. Suggestion is to add that to `RelationPlanner` - The various `table_at` functions are generalised to use the table request instead - The resolution result is also generalised. Same decision to be made here wrt a trait object vs an extensible enum To keep this ergonomic, I'm leaning towards the enum-with-extension approach. If we start from the current state of this PR, that would mean we rename `TableVersion` to something more generic, keep the existing time and version variants, and add a third user-defined one. In the current PR the resolution request and result representation have the same shape; both are `TableVersion`. That should probably be split in two. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
