judahrand commented on issue #12: URL: https://github.com/apache/arrow-datafusion-python/issues/12#issuecomment-1977234377
This is something that would be absolutely fantastic. DuckDB enables this kind of a workflow and we've found it really valuable. The `ibis` API for Datafusion [pretends](https://github.com/ibis-project/ibis/blob/9a741f71b140ef75354b5971930554229c2627df/ibis/backends/datafusion/__init__.py#L490) to do it too... but in reality the `.collect()` call buffers the whole query result into memory so saves nothing... It'd be nice to improve this! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
