Lantao Jin created SPARK-32994:
----------------------------------
Summary: External accumulators (not start with
InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem
Key: SPARK-32994
URL: https://issues.apache.org/jira/browse/SPARK-32994
Project: Spark
Issue Type: Bug
Components: Spark Core, SQL
Affects Versions: 3.0.1, 2.4.7, 3.1.0
Reporter: Lantao Jin
We use Spark + Delta Lake, recently we find our Spark driver faced full GC
problem (very heavy) when users submit a MERGE INTO query. The driver held over
100GB memory (depends on how much the max heap size set) and can not be GC
forever. By making a heap dump we found the root cause.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
