zhztheplayer commented on issue #4392: URL: https://github.com/apache/incubator-gluten/issues/4392#issuecomment-3389812192
@PHILO-HE > when whole stage fallback occurs, the off-heap memory is reduced to 10% of the original value. @FelixYBW > Onheap demand is still very large for stages fallback to vanilla spark. Not sure if I understand correctly, but did you two tend to describe the same phenomenon? Where off-heap usage drops and on-heap usage rises, when the stage falls back? IIRC Spark sort-based shuffle is a heavy operator that remains on on-heap when off-heap is on. I would be glad to do the some path-findings to see if we can somehow fix this. @FelixYBW If you want to also help locate where the remaining on-heap consumption came from? E.g., Could try setting `spark.shuffle.sort.bypassMergeThreshold = 2147483647` to disable vanilla sort-based shuffle then see if the on-heap consumption of vanilla Spark can be reduced? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
