[
https://issues.apache.org/jira/browse/SPARK-52725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dejiu Lu updated SPARK-52725:
-----------------------------
Description:
When we modify the memory configuration in the plugin, the new configuration
cannot be updated to the defaultResourceProfile because it is initialized
before the plugin is loaded.
For example, the initial spark.executor.memory is set to 8g, but after our
plugin processes it, the value is adjusted to 2g. However, the executor's JVM
launch parameter (-Xmx) still defaults to 8g according resource profile, which
could lead to OOM kills due to onheap memory RSS usage exceeds expectations.
Maybe we should adjust the order of Plugin loading and ResourceProfileManager
initialization
was:
When we modify the memory configuration in the plugin, the new configuration
cannot be updated to the defaultResourceProfile because it is initialized
before the plugin is loaded.
For example, the initial spark.executor.memory is set to 8g, but after our
plugin processes it, the value is adjusted to 2g. However, the executor's JVM
launch parameter (-Xmx) still defaults to 8g according resource profile, which
could lead to OOM kills due to onheap memory RSS usage exceeds expectations.
Maybe we should adjust the order of Plugin loading and ResourceProfileManager
initialization
> Using configuration after DriverPlugin loaded to build defaultResourceProfile
> -----------------------------------------------------------------------------
>
> Key: SPARK-52725
> URL: https://issues.apache.org/jira/browse/SPARK-52725
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 4.0.0
> Reporter: Dejiu Lu
> Priority: Minor
>
> When we modify the memory configuration in the plugin, the new configuration
> cannot be updated to the defaultResourceProfile because it is initialized
> before the plugin is loaded.
>
> For example, the initial spark.executor.memory is set to 8g, but after our
> plugin processes it, the value is adjusted to 2g. However, the executor's JVM
> launch parameter (-Xmx) still defaults to 8g according resource profile,
> which could lead to OOM kills due to onheap memory RSS usage exceeds
> expectations.
> Maybe we should adjust the order of Plugin loading and ResourceProfileManager
> initialization
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
