yabola commented on PR #44949: URL: https://github.com/apache/spark/pull/44949#issuecomment-1916225226
I tested a scenario (a large number of broadcast join, lots of iceberg expire snapshot & remove_orphan_files SQL executed concurrently), and found its effect to be very noticeable. On the left side, the job has the parameter enabled, and the job can run smoothly to completion, with memory consumption stable at 40%. On the right side, the parameter is disabled, and it runs out of memory (OOM) after an hour. <img width="1696" alt="企业微信截图_405acd01-1d2f-40af-97c2-034093849ea5" src="https://github.com/apache/spark/assets/31469905/866b8b38-36b5-4d8c-858d-6fe74e77943d";> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
