Terry Wang created SPARK-50421:
----------------------------------
Summary: Executor memory config is incorrect when multiple stage
level resource profiles works
Key: SPARK-50421
URL: https://issues.apache.org/jira/browse/SPARK-50421
Project: Spark
Issue Type: Bug
Components: Scheduler, YARN
Affects Versions: 3.5.3, 3.4.4, 3.3.4, 3.2.4, 3.1.3
Reporter: Terry Wang
https://issues.apache.org/jira/browse/SPARK-27495 has imported a great feature
to help adjust stage's level resource profile(abbv rp), we are observing some
config is not as expected when multiple resource profile works.
When one stage assigned rp2 different with default rp, the
`UnifiedMemoryManager` will still get old conf as the default rp, which will
cause maxOffHeapMemory is still the old value!
{code:scala}
protected[this] val maxOffHeapMemory = conf.get(MEMORY_OFFHEAP_SIZE)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
