CodingCat commented on code in PR #53190:
URL: https://github.com/apache/spark/pull/53190#discussion_r2596966885
##########
core/src/main/scala/org/apache/spark/resource/ResourceProfile.scala:
##########
@@ -48,7 +51,7 @@ import org.apache.spark.util.Utils
@Evolving
@Since("3.1.0")
class ResourceProfile(
- val executorResources: Map[String, ExecutorResourceRequest],
+ var executorResources: Map[String, ExecutorResourceRequest],
Review Comment:
Hi, @mridulm it's a good question, the major reason to make it mutable is
actually for UI and eventLogs
when users do not set memoryOverhead, spark.executor.memoryOverhead will not
show up in UI and event logs, as a result, it brings many troubles for the
monitoring and saving result aggregation.
The way we address this is to add the missing memoryoverhead to resource
profile explicitly at
https://github.com/apache/spark/pull/53190/files#diff-0d090a3c45f4aa7e0959965347f84ab20aeca4051671d0feb0dc05577ca31aa4R73-R93
which requires us to make `executorResources` mutable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
