[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem
[
https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lantao Jin updated SPARK-32994:
---
Description:
We use Spark + Delta Lake, recently we find our Spark driver faced full GC
problem (very heavy) when users submit a MERGE INTO query. The driver held over
100GB memory (depends on how much the max heap size set) and can not be GC
forever. By making a heap dump we found the root cause.
!Screen Shot 2020-09-25 at 11.32.51 AM.png|width=70%!
!Screen Shot 2020-09-25 at 11.35.01 AM.png|width=100%!
!Screen Shot 2020-09-25 at 11.36.48 AM.png|width=100%!
>From above heap dump, Delta uses a SetAccumulator to records touched files
>names
{code}
// Accumulator to collect all the distinct touched files
val touchedFilesAccum = new SetAccumulator[String]()
spark.sparkContext.register(touchedFilesAccum, TOUCHED_FILES_ACCUM_NAME)
// UDFs to records touched files names and add them to the accumulator
val recordTouchedFileName = udf { (fileName: String) => {
touchedFilesAccum.add(fileName)
1
}}.asNondeterministic()
{code}
In a big query, each task may hold thousands of file names, and if a stage
contains dozens of thousands of tasks, DAGscheduler may hold millions of
`CompletionEvent`. And each `CompletionEvent` holds the thousands of file names
in its `accumUpdates`. All accumulator objects will use Spark listener event to
deliver to the event loop and even a full GC can not release memory.
A PR will be submitted. With the patch, the memory problem was gone.
Before the patch: A full GC doesn't help.
!Screen Shot 2020-09-24 at 5.19.58 PM.png|width=70%!
After the patch: No full GC and memory is not ramp up.
!Screen Shot 2020-09-24 at 5.19.26 PM.png|width=70%!
was:
We use Spark + Delta Lake, recently we find our Spark driver faced full GC
problem (very heavy) when users submit a MERGE INTO query. The driver held over
100GB memory (depends on how much the max heap size set) and can not be GC
forever. By making a heap dump we found the root cause.
!Screen Shot 2020-09-25 at 11.32.51 AM.png|width=70%!
!Screen Shot 2020-09-25 at 11.35.01 AM.png|width=100%!
!Screen Shot 2020-09-25 at 11.36.48 AM.png|width=100%!
>From above heap dump, Delta uses a SetAccumulator to records touched files
>names
{code}
// Accumulator to collect all the distinct touched files
val touchedFilesAccum = new SetAccumulator[String]()
spark.sparkContext.register(touchedFilesAccum, TOUCHED_FILES_ACCUM_NAME)
// UDFs to records touched files names and add them to the accumulator
val recordTouchedFileName = udf { (fileName: String) => {
touchedFilesAccum.add(fileName)
1
}}.asNondeterministic()
{code}
In a big query, each task may hold thousands of file names, and if a stage
contains dozens of thousands of tasks, DAGscheduler may hold millions of
`CompletionEvent`. And each `CompletionEvent` holds the thousands of file names
in its `accumUpdates`. All accumulator objects will use Spark listener event to
deliver to the event loop and even a full GC can not release memory.
A PR will be submitted. With the patch, the memory problem was gone.
Before the patch:
!Screen Shot 2020-09-24 at 5.19.58 PM.png|width=70%!
After the patch:
!Screen Shot 2020-09-24 at 5.19.26 PM.png|width=70%!
> External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may
> lead driver full GC problem
> -
>
> Key: SPARK-32994
> URL: https://issues.apache.org/jira/browse/SPARK-32994
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, SQL
>Affects Versions: 2.4.7, 3.0.1, 3.1.0
>Reporter: Lantao Jin
>Priority: Major
> Attachments: Screen Shot 2020-09-24 at 5.19.26 PM.png, Screen Shot
> 2020-09-24 at 5.19.58 PM.png, Screen Shot 2020-09-25 at 11.32.51 AM.png,
> Screen Shot 2020-09-25 at 11.35.01 AM.png, Screen Shot 2020-09-25 at 11.36.48
> AM.png
>
>
> We use Spark + Delta Lake, recently we find our Spark driver faced full GC
> problem (very heavy) when users submit a MERGE INTO query. The driver held
> over 100GB memory (depends on how much the max heap size set) and can not be
> GC forever. By making a heap dump we found the root cause.
> !Screen Shot 2020-09-25 at 11.32.51 AM.png|width=70%!
> !Screen Shot 2020-09-25 at 11.35.01 AM.png|width=100%!
> !Screen Shot 2020-09-25 at 11.36.48 AM.png|width=100%!
> From above heap dump, Delta uses a SetAccumulator to records touched files
> names
> {code}
> // Accumulator to collect all the distinct touched files
> val touchedFilesAccum = new SetAccumulator[String]()
> spark.sparkContext.register(touchedFilesAccum, TOUCHED_FILES_ACCUM_NAME)
> // UDFs to records touched files
[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem
[
https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lantao Jin updated SPARK-32994:
---
Description:
We use Spark + Delta Lake, recently we find our Spark driver faced full GC
problem (very heavy) when users submit a MERGE INTO query. The driver held over
100GB memory (depends on how much the max heap size set) and can not be GC
forever. By making a heap dump we found the root cause.
!Screen Shot 2020-09-25 at 11.32.51 AM.png|width=70%!
!Screen Shot 2020-09-25 at 11.35.01 AM.png|width=100%!
!Screen Shot 2020-09-25 at 11.36.48 AM.png|width=100%!
>From above heap dump, Delta uses a SetAccumulator to records touched files
>names
{code}
// Accumulator to collect all the distinct touched files
val touchedFilesAccum = new SetAccumulator[String]()
spark.sparkContext.register(touchedFilesAccum, TOUCHED_FILES_ACCUM_NAME)
// UDFs to records touched files names and add them to the accumulator
val recordTouchedFileName = udf { (fileName: String) => {
touchedFilesAccum.add(fileName)
1
}}.asNondeterministic()
{code}
In a big query, each task may hold thousands of file names, and if a stage
contains dozens of thousands of tasks, DAGscheduler may hold millions of
`CompletionEvent`. And each `CompletionEvent` holds the thousands of file names
in its `accumUpdates`. All accumulator objects will use Spark listener event to
deliver to the event loop and even a full GC can not release memory.
A PR will be submitted. With the patch, the memory problem was gone.
Before the patch:
!Screen Shot 2020-09-24 at 5.19.58 PM.png|width=70%!
After the patch:
!Screen Shot 2020-09-24 at 5.19.26 PM.png|width=70%!
was:
We use Spark + Delta Lake, recently we find our Spark driver faced full GC
problem (very heavy) when users submit a MERGE INTO query. The driver held over
100GB memory (depends on how much the max heap size set) and can not be GC
forever. By making a heap dump we found the root cause.
!Screen Shot 2020-09-25 at 11.32.51 AM.png|width=70%!
!Screen Shot 2020-09-25 at 11.35.01 AM.png|width=100%!
!Screen Shot 2020-09-25 at 11.36.48 AM.png|width=100%!
> External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may
> lead driver full GC problem
> -
>
> Key: SPARK-32994
> URL: https://issues.apache.org/jira/browse/SPARK-32994
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, SQL
>Affects Versions: 2.4.7, 3.0.1, 3.1.0
>Reporter: Lantao Jin
>Priority: Major
> Attachments: Screen Shot 2020-09-24 at 5.19.26 PM.png, Screen Shot
> 2020-09-24 at 5.19.58 PM.png, Screen Shot 2020-09-25 at 11.32.51 AM.png,
> Screen Shot 2020-09-25 at 11.35.01 AM.png, Screen Shot 2020-09-25 at 11.36.48
> AM.png
>
>
> We use Spark + Delta Lake, recently we find our Spark driver faced full GC
> problem (very heavy) when users submit a MERGE INTO query. The driver held
> over 100GB memory (depends on how much the max heap size set) and can not be
> GC forever. By making a heap dump we found the root cause.
> !Screen Shot 2020-09-25 at 11.32.51 AM.png|width=70%!
> !Screen Shot 2020-09-25 at 11.35.01 AM.png|width=100%!
> !Screen Shot 2020-09-25 at 11.36.48 AM.png|width=100%!
> From above heap dump, Delta uses a SetAccumulator to records touched files
> names
> {code}
> // Accumulator to collect all the distinct touched files
> val touchedFilesAccum = new SetAccumulator[String]()
> spark.sparkContext.register(touchedFilesAccum, TOUCHED_FILES_ACCUM_NAME)
> // UDFs to records touched files names and add them to the accumulator
> val recordTouchedFileName = udf { (fileName: String) => {
> touchedFilesAccum.add(fileName)
> 1
> }}.asNondeterministic()
> {code}
> In a big query, each task may hold thousands of file names, and if a stage
> contains dozens of thousands of tasks, DAGscheduler may hold millions of
> `CompletionEvent`. And each `CompletionEvent` holds the thousands of file
> names in its `accumUpdates`. All accumulator objects will use Spark listener
> event to deliver to the event loop and even a full GC can not release memory.
> A PR will be submitted. With the patch, the memory problem was gone.
> Before the patch:
> !Screen Shot 2020-09-24 at 5.19.58 PM.png|width=70%!
> After the patch:
> !Screen Shot 2020-09-24 at 5.19.26 PM.png|width=70%!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem
[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Attachment: Screen Shot 2020-09-24 at 5.19.26 PM.png > External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may > lead driver full GC problem > - > > Key: SPARK-32994 > URL: https://issues.apache.org/jira/browse/SPARK-32994 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 2.4.7, 3.0.1, 3.1.0 >Reporter: Lantao Jin >Priority: Major > Attachments: Screen Shot 2020-09-24 at 5.19.26 PM.png, Screen Shot > 2020-09-24 at 5.19.58 PM.png, Screen Shot 2020-09-25 at 11.32.51 AM.png, > Screen Shot 2020-09-25 at 11.35.01 AM.png, Screen Shot 2020-09-25 at 11.36.48 > AM.png > > > We use Spark + Delta Lake, recently we find our Spark driver faced full GC > problem (very heavy) when users submit a MERGE INTO query. The driver held > over 100GB memory (depends on how much the max heap size set) and can not be > GC forever. By making a heap dump we found the root cause. > !Screen Shot 2020-09-25 at 11.32.51 AM.png|width=70%! > !Screen Shot 2020-09-25 at 11.35.01 AM.png|width=100%! > !Screen Shot 2020-09-25 at 11.36.48 AM.png|width=100%! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem
[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Attachment: Screen Shot 2020-09-24 at 5.19.58 PM.png > External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may > lead driver full GC problem > - > > Key: SPARK-32994 > URL: https://issues.apache.org/jira/browse/SPARK-32994 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 2.4.7, 3.0.1, 3.1.0 >Reporter: Lantao Jin >Priority: Major > Attachments: Screen Shot 2020-09-24 at 5.19.58 PM.png, Screen Shot > 2020-09-25 at 11.32.51 AM.png, Screen Shot 2020-09-25 at 11.35.01 AM.png, > Screen Shot 2020-09-25 at 11.36.48 AM.png > > > We use Spark + Delta Lake, recently we find our Spark driver faced full GC > problem (very heavy) when users submit a MERGE INTO query. The driver held > over 100GB memory (depends on how much the max heap size set) and can not be > GC forever. By making a heap dump we found the root cause. > !Screen Shot 2020-09-25 at 11.32.51 AM.png|width=70%! > !Screen Shot 2020-09-25 at 11.35.01 AM.png|width=100%! > !Screen Shot 2020-09-25 at 11.36.48 AM.png|width=100%! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem
[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Description: We use Spark + Delta Lake, recently we find our Spark driver faced full GC problem (very heavy) when users submit a MERGE INTO query. The driver held over 100GB memory (depends on how much the max heap size set) and can not be GC forever. By making a heap dump we found the root cause. !Screen Shot 2020-09-25 at 11.32.51 AM.png|width=70%! !Screen Shot 2020-09-25 at 11.35.01 AM.png|width=70%! !Screen Shot 2020-09-25 at 11.36.48 AM.png|width=70%! was: We use Spark + Delta Lake, recently we find our Spark driver faced full GC problem (very heavy) when users submit a MERGE INTO query. The driver held over 100GB memory (depends on how much the max heap size set) and can not be GC forever. By making a heap dump we found the root cause. !Screen Shot 2020-09-25 at 11.32.51 AM.png|width=100%! > External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may > lead driver full GC problem > - > > Key: SPARK-32994 > URL: https://issues.apache.org/jira/browse/SPARK-32994 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 2.4.7, 3.0.1, 3.1.0 >Reporter: Lantao Jin >Priority: Major > Attachments: Screen Shot 2020-09-25 at 11.32.51 AM.png, Screen Shot > 2020-09-25 at 11.35.01 AM.png, Screen Shot 2020-09-25 at 11.36.48 AM.png > > > We use Spark + Delta Lake, recently we find our Spark driver faced full GC > problem (very heavy) when users submit a MERGE INTO query. The driver held > over 100GB memory (depends on how much the max heap size set) and can not be > GC forever. By making a heap dump we found the root cause. > !Screen Shot 2020-09-25 at 11.32.51 AM.png|width=70%! > !Screen Shot 2020-09-25 at 11.35.01 AM.png|width=70%! > !Screen Shot 2020-09-25 at 11.36.48 AM.png|width=70%! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem
[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Description: We use Spark + Delta Lake, recently we find our Spark driver faced full GC problem (very heavy) when users submit a MERGE INTO query. The driver held over 100GB memory (depends on how much the max heap size set) and can not be GC forever. By making a heap dump we found the root cause. !Screen Shot 2020-09-25 at 11.32.51 AM.png|width=70%! !Screen Shot 2020-09-25 at 11.35.01 AM.png|width=100%! !Screen Shot 2020-09-25 at 11.36.48 AM.png|width=100%! was: We use Spark + Delta Lake, recently we find our Spark driver faced full GC problem (very heavy) when users submit a MERGE INTO query. The driver held over 100GB memory (depends on how much the max heap size set) and can not be GC forever. By making a heap dump we found the root cause. !Screen Shot 2020-09-25 at 11.32.51 AM.png|width=70%! !Screen Shot 2020-09-25 at 11.35.01 AM.png|width=70%! !Screen Shot 2020-09-25 at 11.36.48 AM.png|width=70%! > External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may > lead driver full GC problem > - > > Key: SPARK-32994 > URL: https://issues.apache.org/jira/browse/SPARK-32994 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 2.4.7, 3.0.1, 3.1.0 >Reporter: Lantao Jin >Priority: Major > Attachments: Screen Shot 2020-09-25 at 11.32.51 AM.png, Screen Shot > 2020-09-25 at 11.35.01 AM.png, Screen Shot 2020-09-25 at 11.36.48 AM.png > > > We use Spark + Delta Lake, recently we find our Spark driver faced full GC > problem (very heavy) when users submit a MERGE INTO query. The driver held > over 100GB memory (depends on how much the max heap size set) and can not be > GC forever. By making a heap dump we found the root cause. > !Screen Shot 2020-09-25 at 11.32.51 AM.png|width=70%! > !Screen Shot 2020-09-25 at 11.35.01 AM.png|width=100%! > !Screen Shot 2020-09-25 at 11.36.48 AM.png|width=100%! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem
[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Description: We use Spark + Delta Lake, recently we find our Spark driver faced full GC problem (very heavy) when users submit a MERGE INTO query. The driver held over 100GB memory (depends on how much the max heap size set) and can not be GC forever. By making a heap dump we found the root cause. !Screen Shot 2020-09-25 at 11.32.51 AM.png|width=100%! was: We use Spark + Delta Lake, recently we find our Spark driver faced full GC problem (very heavy) when users submit a MERGE INTO query. The driver held over 100GB memory (depends on how much the max heap size set) and can not be GC forever. By making a heap dump we found the root cause. !Screen Shot 2020-09-25 at 11.32.51 AM.png | width=100%! > External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may > lead driver full GC problem > - > > Key: SPARK-32994 > URL: https://issues.apache.org/jira/browse/SPARK-32994 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 2.4.7, 3.0.1, 3.1.0 >Reporter: Lantao Jin >Priority: Major > Attachments: Screen Shot 2020-09-25 at 11.32.51 AM.png, Screen Shot > 2020-09-25 at 11.35.01 AM.png, Screen Shot 2020-09-25 at 11.36.48 AM.png > > > We use Spark + Delta Lake, recently we find our Spark driver faced full GC > problem (very heavy) when users submit a MERGE INTO query. The driver held > over 100GB memory (depends on how much the max heap size set) and can not be > GC forever. By making a heap dump we found the root cause. > !Screen Shot 2020-09-25 at 11.32.51 AM.png|width=100%! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem
[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Description: We use Spark + Delta Lake, recently we find our Spark driver faced full GC problem (very heavy) when users submit a MERGE INTO query. The driver held over 100GB memory (depends on how much the max heap size set) and can not be GC forever. By making a heap dump we found the root cause. !Screen Shot 2020-09-25 at 11.32.51 AM.png! was: We use Spark + Delta Lake, recently we find our Spark driver faced full GC problem (very heavy) when users submit a MERGE INTO query. The driver held over 100GB memory (depends on how much the max heap size set) and can not be GC forever. By making a heap dump we found the root cause. > External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may > lead driver full GC problem > - > > Key: SPARK-32994 > URL: https://issues.apache.org/jira/browse/SPARK-32994 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 2.4.7, 3.0.1, 3.1.0 >Reporter: Lantao Jin >Priority: Major > Attachments: Screen Shot 2020-09-25 at 11.32.51 AM.png, Screen Shot > 2020-09-25 at 11.35.01 AM.png, Screen Shot 2020-09-25 at 11.36.48 AM.png > > > We use Spark + Delta Lake, recently we find our Spark driver faced full GC > problem (very heavy) when users submit a MERGE INTO query. The driver held > over 100GB memory (depends on how much the max heap size set) and can not be > GC forever. By making a heap dump we found the root cause. > !Screen Shot 2020-09-25 at 11.32.51 AM.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem
[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Attachment: Screen Shot 2020-09-25 at 11.36.48 AM.png > External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may > lead driver full GC problem > - > > Key: SPARK-32994 > URL: https://issues.apache.org/jira/browse/SPARK-32994 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 2.4.7, 3.0.1, 3.1.0 >Reporter: Lantao Jin >Priority: Major > Attachments: Screen Shot 2020-09-25 at 11.32.51 AM.png, Screen Shot > 2020-09-25 at 11.35.01 AM.png, Screen Shot 2020-09-25 at 11.36.48 AM.png > > > We use Spark + Delta Lake, recently we find our Spark driver faced full GC > problem (very heavy) when users submit a MERGE INTO query. The driver held > over 100GB memory (depends on how much the max heap size set) and can not be > GC forever. By making a heap dump we found the root cause. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem
[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Attachment: Screen Shot 2020-09-25 at 11.32.51 AM.png > External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may > lead driver full GC problem > - > > Key: SPARK-32994 > URL: https://issues.apache.org/jira/browse/SPARK-32994 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 2.4.7, 3.0.1, 3.1.0 >Reporter: Lantao Jin >Priority: Major > Attachments: Screen Shot 2020-09-25 at 11.32.51 AM.png, Screen Shot > 2020-09-25 at 11.35.01 AM.png, Screen Shot 2020-09-25 at 11.36.48 AM.png > > > We use Spark + Delta Lake, recently we find our Spark driver faced full GC > problem (very heavy) when users submit a MERGE INTO query. The driver held > over 100GB memory (depends on how much the max heap size set) and can not be > GC forever. By making a heap dump we found the root cause. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem
[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Attachment: Screen Shot 2020-09-25 at 11.35.01 AM.png > External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may > lead driver full GC problem > - > > Key: SPARK-32994 > URL: https://issues.apache.org/jira/browse/SPARK-32994 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 2.4.7, 3.0.1, 3.1.0 >Reporter: Lantao Jin >Priority: Major > Attachments: Screen Shot 2020-09-25 at 11.32.51 AM.png, Screen Shot > 2020-09-25 at 11.35.01 AM.png, Screen Shot 2020-09-25 at 11.36.48 AM.png > > > We use Spark + Delta Lake, recently we find our Spark driver faced full GC > problem (very heavy) when users submit a MERGE INTO query. The driver held > over 100GB memory (depends on how much the max heap size set) and can not be > GC forever. By making a heap dump we found the root cause. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem
[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Description: We use Spark + Delta Lake, recently we find our Spark driver faced full GC problem (very heavy) when users submit a MERGE INTO query. The driver held over 100GB memory (depends on how much the max heap size set) and can not be GC forever. By making a heap dump we found the root cause. !Screen Shot 2020-09-25 at 11.32.51 AM.png | width=100%! was: We use Spark + Delta Lake, recently we find our Spark driver faced full GC problem (very heavy) when users submit a MERGE INTO query. The driver held over 100GB memory (depends on how much the max heap size set) and can not be GC forever. By making a heap dump we found the root cause. !Screen Shot 2020-09-25 at 11.32.51 AM.png! > External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may > lead driver full GC problem > - > > Key: SPARK-32994 > URL: https://issues.apache.org/jira/browse/SPARK-32994 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 2.4.7, 3.0.1, 3.1.0 >Reporter: Lantao Jin >Priority: Major > Attachments: Screen Shot 2020-09-25 at 11.32.51 AM.png, Screen Shot > 2020-09-25 at 11.35.01 AM.png, Screen Shot 2020-09-25 at 11.36.48 AM.png > > > We use Spark + Delta Lake, recently we find our Spark driver faced full GC > problem (very heavy) when users submit a MERGE INTO query. The driver held > over 100GB memory (depends on how much the max heap size set) and can not be > GC forever. By making a heap dump we found the root cause. > !Screen Shot 2020-09-25 at 11.32.51 AM.png | width=100%! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
