westonpace commented on PR #9015: URL: https://github.com/apache/arrow-datafusion/pull/9015#issuecomment-2027533840
Yes, I'll chime in a bit as I did end up finding the root issue. My plan had both `SortExec` and `SortPreservingMergeExec` and I had incorrectly assumed `SortPreservingMergeExec` was not a spillable operator and so I was confused why I was getting out-of-memory errors. Having a tool like you've shown that would help to visualize memory usage would have allowed me to figure that out more quickly. Also, we have some customers that want to know, in great detail, why various operations need RAM and what the limits of that RAM will be. I think this could be just a start into greater RAM observability. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
