PHILO-HE commented on issue #4392: URL: https://github.com/apache/incubator-gluten/issues/4392#issuecomment-3388770568
> Maybe it's known feature. We encountered a case in Spark3.5. Once you enable offheap in Spark, Spark will try to use offheap to cache the internalrows, if offheap memory isn't enough, it won't check the onheap memory but just spill the data. So if you config a large on heap memory but small offheap memory, spill will be triggered more frequently. > > While you either can't config a large offheap memory and a small onheap memory because not all operators are able to cache the internal rows into offheap. Onheap demand is still very large for stages fallback to vanilla spark. @FelixYBW, we didn't realize this issue before. In Gluten's current resource profile rule, when whole stage fallback occurs, the offheap memory is reduced to 10% of the original value. Then, if I understand correctly, this setting can cause perf regression. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
