adriangb commented on PR #22000:
URL: https://github.com/apache/datafusion/pull/22000#issuecomment-4374101528
> Did you try and implement whatever sampling you need with just the
existing APIs? Aka no change to the core? If that wasn't possible, what was
missing in the API?
Yes, it won't work sadly. With current APIs you could maybe sample at the
file level, but not at the row group or page level (which requires reading
parquet footers for all files, parsing them, etc). The way I implemented things
here that is lazily deferred to file opening so there is no wasted work.
It would also require a lot of tight coupling (mainly via downcast
matching) between the custom optimizer rule, the custom physical plan and the
custom data source. The way I've set it up in this PR if e.g. Vortex wanted to
implement table sampling they could do that in a self contained way inside of
their file source implementation quite easily.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
