AngersZhuuuu opened a new pull request, #57716: URL: https://github.com/apache/spark/pull/57716
### What changes were proposed in this pull request? This patch updates UnifiedMemoryManager to validate memory according to the current process role. - SparkEnv passes the current process role to UnifiedMemoryManager. - Driver validates Driver JVM memory only. - Executor validates spark.executor.memory only. ### Why are the changes needed? When spark.executor.memory is set to 500m, the Executor may fail with the following error: ``` Exception in thread "main" java.lang.IllegalArgumentException: System memory 466092032 must be at least 471859200. Please increase heap size using the --driver-memory option or spark.driver.memory in Spark configuration. ``` The error is misleading because UnifiedMemoryManager is initialized in the Executor, but the shared validation logic checks the runtime system memory as Driver memory and reports Driver-specific configuration guidance. The Executor should validate its own executor memory configuration instead of applying the Driver memory check. ### Does this PR introduce any user-facing change? Yes. Previously, an Executor could report a Driver memory error when its runtime system memory was below the Driver threshold. After this change: - Driver-side memory failures continue to report Driver memory configuration guidance. - Executor-side memory failures validate and report Executor memory configuration. - An Executor with sufficient spark.executor.memory will not fail because of the Driver heap validation. ### How was this patch tested? Added UT ### Was this patch authored or co-authored using generative AI tooling? Yea -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
