Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/7770#discussion_r36047313
--- Diff: core/src/main/scala/org/apache/spark/ui/ToolTips.scala ---
@@ -62,6 +62,13 @@ private[spark] object ToolTips {
"""Time that the executor spent paused for Java garbage collection
while the task was
running."""
+ val PEAK_EXECUTION_MEMORY =
+ """Execution memory refers to the memory used by internal data
structures created during
+ shuffles, aggregations and joins when Tungsten is enabled. The
value of this accumulator
+ should be approximately the sum of the peak sizes across all such
data structures created
+ in this task. For SQL jobs, this only tracks all unsafe operators,
broadcast joins, and
+ external sort."""
--- End diff --
I just commented on this elsewhere, but I have one concern about disabling
this metric when Tungsten is disabled: users may run non-SQL/DataFrame jobs on
a cluster where Tungsten has been enabled, so feature-flagging this based on
whether Tungsten is on doesn't necessarily solve this confusion due to
under-accounting of non-tracked memory.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
