lidavidm commented on PR #59313: URL: https://github.com/apache/spark/pull/59313#issuecomment-6158610570
Pushdown semantics has come up quite a few times (e.g. @SamArch27 evaluating how to do pushdown for DuckDB-ADBC). I think it would require something like Substrait so that semantics are consistent across engines and drivers, with each driver supporting a limited subset (e.g. table scans with filters) and translating to SQL if needed. ADBC does do streaming reads, and some drivers can even expose a read as a set of partitions that can be distributed across workers. As for reads with offsets, that could perhaps be done via Substrait as well, if we go that direction. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
