Dejiu Lu created SPARK-52725:
--------------------------------
Summary: Using configuration after DriverPlugin loaded to build
defaultResourceProfile
Key: SPARK-52725
URL: https://issues.apache.org/jira/browse/SPARK-52725
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 4.0.0
Reporter: Dejiu Lu
When we modify the memory configuration in the plugin, the new configuration
cannot be updated to the defaultResourceProfile because it is initialized
before the plugin is loaded.
For example, the initial spark.executor.memory is set to 8g, but after our
plugin processes it, the value is adjusted to 2g. However, the executor's JVM
launch parameter (-Xmx) still defaults to 8g according resource profile, which
could lead to OOM kills due to onheap memory RSS usage exceeds expectations.
Maybe we should adjust the order of Plugin loading and ResourceProfileManager
initialization
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
