edrevo commented on pull request #9523: URL: https://github.com/apache/arrow/pull/9523#issuecomment-786180030
@andygrove / @seddonm1, I think we should revert this: RepartitionExec will now load everything into memory, since we are now .awaiting untill the input stream has been completely consumed, which moves all of the data into memory. Maybe it would be worth understanding the root cause of the original bug to understand what is going on with the last partition. I had a similar behavior with ballista recently and it was due to https://issues.apache.org/jira/browse/ARROW-11784 ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
