[
https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lantao Jin updated SPARK-32994:
-------------------------------
Summary: Heavy external accumulators may lead driver full GC problem (was:
External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may
lead driver full GC problem)
> Heavy external accumulators may lead driver full GC problem
> -----------------------------------------------------------
>
> Key: SPARK-32994
> URL: https://issues.apache.org/jira/browse/SPARK-32994
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, SQL
> Affects Versions: 2.4.7, 3.0.1, 3.1.0
> Reporter: Lantao Jin
> Priority: Major
> Attachments: Screen Shot 2020-09-24 at 5.19.26 PM.png, Screen Shot
> 2020-09-24 at 5.19.58 PM.png, Screen Shot 2020-09-25 at 11.32.51 AM.png,
> Screen Shot 2020-09-25 at 11.35.01 AM.png, Screen Shot 2020-09-25 at 11.36.48
> AM.png
>
>
> We use Spark + Delta Lake, recently we find our Spark driver faced full GC
> problem (very heavy) when users submit a MERGE INTO query. The driver held
> over 100GB memory (depends on how much the max heap size set) and can not be
> GC forever. By making a heap dump we found the root cause.
> !Screen Shot 2020-09-25 at 11.32.51 AM.png|width=70%!
> !Screen Shot 2020-09-25 at 11.35.01 AM.png|width=100%!
> !Screen Shot 2020-09-25 at 11.36.48 AM.png|width=100%!
> From above heap dump, Delta uses a SetAccumulator to records touched files
> names
> {code}
> // Accumulator to collect all the distinct touched files
> val touchedFilesAccum = new SetAccumulator[String]()
> spark.sparkContext.register(touchedFilesAccum, TOUCHED_FILES_ACCUM_NAME)
> // UDFs to records touched files names and add them to the accumulator
> val recordTouchedFileName = udf { (fileName: String) => {
> touchedFilesAccum.add(fileName)
> 1
> }}.asNondeterministic()
> {code}
> In a big query, each task may hold thousands of file names, and if a stage
> contains dozens of thousands of tasks, DAGscheduler may hold millions of
> `CompletionEvent`. And each `CompletionEvent` holds the thousands of file
> names in its `accumUpdates`. All accumulator objects will use Spark listener
> event to deliver to the event loop and even a full GC can not release memory.
> A PR will be submitted. With the patch, the memory problem was gone.
> Before the patch: A full GC doesn't help.
> !Screen Shot 2020-09-24 at 5.19.58 PM.png|width=70%!
> After the patch: No full GC and memory is not ramp up.
> !Screen Shot 2020-09-24 at 5.19.26 PM.png|width=70%!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
