Kontinuation commented on PR #1021: URL: https://github.com/apache/datafusion-comet/pull/1021#issuecomment-2419719313
> Thanks for filing this @Kontinuation. I will close my PRs. > > Also, there is a suggestion in #1017 for always using unified memory. I agree that using the unified memory manager is a better approach. Vanilla Spark operators and comet operators are governed by the same memory manager and they are all using offheap memory. Vanilla Spark operators can free some memory when comet operators are under pressure. I'll also put more work into improving the unified memory management. I think the native memory management approach may still be relevant when users don't want vanilla Spark to use off-heap memory. We can set the default value of `spark.comet.exec.memoryPool` as `greedy_task_shared` to make comet memory overhead configuration more intuitive. If users want memory-intensive operators to spill properly, they can try out `fair_spill_task_shared`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
