FelixYBW commented on issue #8025: URL: https://github.com/apache/incubator-gluten/issues/8025#issuecomment-2495309752
Looks it's the same issue as shuffle spill. All spill merge should have the same issue. we should solve it by similar way. 1) is the tradeoff of spill file# and overhead memory. 2) I'd think 4M is too large. What's the vanilla spark's spill buffer size? is it configurable? in theory vanilla spark has the same issue as Gluten. @jinchengchenghh do you know? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
