FelixYBW commented on issue #3030: URL: https://github.com/apache/incubator-gluten/issues/3030#issuecomment-2540309645
#6556 The root cause is that an operator already trigger spill, but later a huge memory allocation is required again. E.g: sort spill the data, later when get_output is called, it allocates huge memory for spilled files to merge output. The allocation either trigger new spill which causes OOM exception or is from global memory which causes kill by Yarn. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
