jpcorreia99 commented on PR #45240: URL: https://github.com/apache/spark/pull/45240#issuecomment-1976891862
@tgravescs very fair point! We preconfigure the spark nodes we provide to users with a global config, but then allow the users to override some of the fields. We observed a reduction in OOMs when bumping the minimum memory overhead. The reason why we don't explicitly set the memory overhead is because we could accidentally be reducing the overall memory the user has access to. As an example, we'd like to have at least 500Mb of overhead. User has requested a 30Gb driver. If we directly set the memory overhead to 500Mb, we'd be cutting the overhead available to the user by 2.5Gb (as overhead memory is calculated as 10% of JVM). We also don't want to be changing the overhead factor since these are global config. If we'd like to have at least a 500Mb overhead and we have a default memory of 2Gb, we'd need to set up the factor at 0.25. But then, if the user comes and asks for a 30Gb driver, he'd also receive 7.5Gb in overhead, which is excessive. As such, defining a global overhead memory minimum would work best if you want to have configurable memory settings and reasonable overhead defaults. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
