Andrew Ash created SPARK-21941:
----------------------------------
Summary: Stop storing unused attemptId in SQLTaskMetrics
Key: SPARK-21941
URL: https://issues.apache.org/jira/browse/SPARK-21941
Project: Spark
Issue Type: Bug
Components: Web UI
Affects Versions: 2.2.0
Reporter: Andrew Ash
Currently SQLTaskMetrics has a long attemptId field on it that is unused, with
a TODO saying to populate the value in the future. We should save this memory
by leaving the TODO but taking the unused field out of the class.
I have a driver that heap dumped on OOM and has 390,105 instances of
SQLTaskMetric -- removing this 8 bytes field will save roughly 390k*8 = 3.1MB
of heap space. It's not going to fix my OOM, but there's no reason to put this
pressure on the GC if we don't get anything by storing it.
https://github.com/apache/spark/blob/v2.2.0/sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLListener.scala#L485
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
