zhztheplayer commented on issue #10932: URL: https://github.com/apache/incubator-gluten/issues/10932#issuecomment-3441940021
> I was wondering if this could potentially be related to the issue? https://github.com/apache/incubator-gluten/commit/083ecfbbe9cbb8d2dffd86c177ac5e1debdac973#diff-b6234f870afb82ba142a4f4e3e358ddb30dc4d5f00b3b9f5b4e9afddc9b4a761R55 @zhli1142015 I was trying to understand why this issue is related to https://github.com/apache/incubator-gluten/pull/9336. As I mentioned, https://github.com/apache/incubator-gluten/pull/9336 was for optimizing the first version of this feature, which was implemented by https://github.com/apache/incubator-gluten/pull/5439. According to that version of the feature, `DynamicOffHeapSizingMemoryTarget` was using exactly the same capacity as Spark's off-heap memory. https://github.com/apache/incubator-gluten/blob/55b2e92f54f95f980587d08f0beb430e0a922b15/gluten-core/src/main/java/org/apache/gluten/memory/memtarget/DynamicOffHeapSizingMemoryTarget.java#L33 So in #9336, we could safely do the simplification as spill was never triggered. I was once thinking about removing the whole feature because it was no longer maintained: https://github.com/apache/incubator-gluten/discussions/8600, #9336 was kind of, at the same time, preparing for that. @zhli1142015 Would you confirm whether we are on the same page about the above? Thanks. And after these actions, the good news was we got the feature reworked by https://github.com/apache/incubator-gluten/pull/9585. Design was changed a lot so I think we can do anything we think is right to make sure spill works for the feature. Maybe by introducing the delegated target back, I am not sure. > By default, spark.memory.fraction is 0.6, meaning that only about 60% of the executor memory (after subtracting 300 MB for overhead) is available for task computation. I would recommend taking more comprehensive factors into consideration when relying on such mechanism for spilling, because we cannot make sure on-heap memory doesn't exceed 40%. Once it exceeds, how can we trigger spill prior to OOM, etc. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
