alamb commented on pull request #1526: URL: https://github.com/apache/arrow-datafusion/pull/1526#issuecomment-1009924630
> I think there is a gap between ExecPlan and MemoryConsumer. Since an execute method would be called multiple times with different partition, it's always the SendableRecordBatchStream such as SortPreservingMergeStream, CrossJoinStream that takes up memory. Should I make it like: This is a good point (that the memory management is done on a per-partition basis rather than a per `ExecutionPlan` basis. I need to think 🤔 about it some more. I would recommend we don't change `SendableRecordBatchStream` which is complicated enough as is. I will make time today to review this PR again thoroughly -- thank you @yjshen I think we are close -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
