wesm commented on issue #7338: URL: https://github.com/apache/arrow/issues/7338#issuecomment-638329728
We're actively working toward having a complete streaming query engine and a low-level data frame API that will permit analytics on filtered, memory-mapped data without having to materialize everything in memory. It has been discussed at length on the mailing list. In the meantime to read datasets into pandas requires materializing them. So I'll close this for now and refer you to further discussions over e-mail ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
