[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem
[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Description: We use Spark + Delta Lake, recently we find our Spark driver faced full GC problem (very heavy) when users submit a MERGE INTO query. The driver held over 100GB memory (depends on how much the max heap size set) and can not be GC forever. By making a heap dump we found the root cause. !Screen Shot 2020-09-25 at 11.32.51 AM.png|width=70%! !Screen Shot 2020-09-25 at 11.35.01 AM.png|width=100%! !Screen Shot 2020-09-25 at 11.36.48 AM.png|width=100%! >From above heap dump, Delta uses a SetAccumulator to records touched files >names {code} // Accumulator to collect all the distinct touched files val touchedFilesAccum = new SetAccumulator[String]() spark.sparkContext.register(touchedFilesAccum, TOUCHED_FILES_ACCUM_NAME) // UDFs to records touched files names and add them to the accumulator val recordTouchedFileName = udf { (fileName: String) => { touchedFilesAccum.add(fileName) 1 }}.asNondeterministic() {code} In a big query, each task may hold thousands of file names, and if a stage contains dozens of thousands of tasks, DAGscheduler may hold millions of `CompletionEvent`. And each `CompletionEvent` holds the thousands of file names in its `accumUpdates`. All accumulator objects will use Spark listener event to deliver to the event loop and even a full GC can not release memory. A PR will be submitted. With the patch, the memory problem was gone. Before the patch: A full GC doesn't help. !Screen Shot 2020-09-24 at 5.19.58 PM.png|width=70%! After the patch: No full GC and memory is not ramp up. !Screen Shot 2020-09-24 at 5.19.26 PM.png|width=70%! was: We use Spark + Delta Lake, recently we find our Spark driver faced full GC problem (very heavy) when users submit a MERGE INTO query. The driver held over 100GB memory (depends on how much the max heap size set) and can not be GC forever. By making a heap dump we found the root cause. !Screen Shot 2020-09-25 at 11.32.51 AM.png|width=70%! !Screen Shot 2020-09-25 at 11.35.01 AM.png|width=100%! !Screen Shot 2020-09-25 at 11.36.48 AM.png|width=100%! >From above heap dump, Delta uses a SetAccumulator to records touched files >names {code} // Accumulator to collect all the distinct touched files val touchedFilesAccum = new SetAccumulator[String]() spark.sparkContext.register(touchedFilesAccum, TOUCHED_FILES_ACCUM_NAME) // UDFs to records touched files names and add them to the accumulator val recordTouchedFileName = udf { (fileName: String) => { touchedFilesAccum.add(fileName) 1 }}.asNondeterministic() {code} In a big query, each task may hold thousands of file names, and if a stage contains dozens of thousands of tasks, DAGscheduler may hold millions of `CompletionEvent`. And each `CompletionEvent` holds the thousands of file names in its `accumUpdates`. All accumulator objects will use Spark listener event to deliver to the event loop and even a full GC can not release memory. A PR will be submitted. With the patch, the memory problem was gone. Before the patch: !Screen Shot 2020-09-24 at 5.19.58 PM.png|width=70%! After the patch: !Screen Shot 2020-09-24 at 5.19.26 PM.png|width=70%! > External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may > lead driver full GC problem > - > > Key: SPARK-32994 > URL: https://issues.apache.org/jira/browse/SPARK-32994 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 2.4.7, 3.0.1, 3.1.0 >Reporter: Lantao Jin >Priority: Major > Attachments: Screen Shot 2020-09-24 at 5.19.26 PM.png, Screen Shot > 2020-09-24 at 5.19.58 PM.png, Screen Shot 2020-09-25 at 11.32.51 AM.png, > Screen Shot 2020-09-25 at 11.35.01 AM.png, Screen Shot 2020-09-25 at 11.36.48 > AM.png > > > We use Spark + Delta Lake, recently we find our Spark driver faced full GC > problem (very heavy) when users submit a MERGE INTO query. The driver held > over 100GB memory (depends on how much the max heap size set) and can not be > GC forever. By making a heap dump we found the root cause. > !Screen Shot 2020-09-25 at 11.32.51 AM.png|width=70%! > !Screen Shot 2020-09-25 at 11.35.01 AM.png|width=100%! > !Screen Shot 2020-09-25 at 11.36.48 AM.png|width=100%! > From above heap dump, Delta uses a SetAccumulator to records touched files > names > {code} > // Accumulator to collect all the distinct touched files > val touchedFilesAccum = new SetAccumulator[String]() > spark.sparkContext.register(touchedFilesAccum, TOUCHED_FILES_ACCUM_NAME) > // UDFs to records touched files
[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem
[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Description: We use Spark + Delta Lake, recently we find our Spark driver faced full GC problem (very heavy) when users submit a MERGE INTO query. The driver held over 100GB memory (depends on how much the max heap size set) and can not be GC forever. By making a heap dump we found the root cause. !Screen Shot 2020-09-25 at 11.32.51 AM.png|width=70%! !Screen Shot 2020-09-25 at 11.35.01 AM.png|width=100%! !Screen Shot 2020-09-25 at 11.36.48 AM.png|width=100%! >From above heap dump, Delta uses a SetAccumulator to records touched files >names {code} // Accumulator to collect all the distinct touched files val touchedFilesAccum = new SetAccumulator[String]() spark.sparkContext.register(touchedFilesAccum, TOUCHED_FILES_ACCUM_NAME) // UDFs to records touched files names and add them to the accumulator val recordTouchedFileName = udf { (fileName: String) => { touchedFilesAccum.add(fileName) 1 }}.asNondeterministic() {code} In a big query, each task may hold thousands of file names, and if a stage contains dozens of thousands of tasks, DAGscheduler may hold millions of `CompletionEvent`. And each `CompletionEvent` holds the thousands of file names in its `accumUpdates`. All accumulator objects will use Spark listener event to deliver to the event loop and even a full GC can not release memory. A PR will be submitted. With the patch, the memory problem was gone. Before the patch: !Screen Shot 2020-09-24 at 5.19.58 PM.png|width=70%! After the patch: !Screen Shot 2020-09-24 at 5.19.26 PM.png|width=70%! was: We use Spark + Delta Lake, recently we find our Spark driver faced full GC problem (very heavy) when users submit a MERGE INTO query. The driver held over 100GB memory (depends on how much the max heap size set) and can not be GC forever. By making a heap dump we found the root cause. !Screen Shot 2020-09-25 at 11.32.51 AM.png|width=70%! !Screen Shot 2020-09-25 at 11.35.01 AM.png|width=100%! !Screen Shot 2020-09-25 at 11.36.48 AM.png|width=100%! > External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may > lead driver full GC problem > - > > Key: SPARK-32994 > URL: https://issues.apache.org/jira/browse/SPARK-32994 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 2.4.7, 3.0.1, 3.1.0 >Reporter: Lantao Jin >Priority: Major > Attachments: Screen Shot 2020-09-24 at 5.19.26 PM.png, Screen Shot > 2020-09-24 at 5.19.58 PM.png, Screen Shot 2020-09-25 at 11.32.51 AM.png, > Screen Shot 2020-09-25 at 11.35.01 AM.png, Screen Shot 2020-09-25 at 11.36.48 > AM.png > > > We use Spark + Delta Lake, recently we find our Spark driver faced full GC > problem (very heavy) when users submit a MERGE INTO query. The driver held > over 100GB memory (depends on how much the max heap size set) and can not be > GC forever. By making a heap dump we found the root cause. > !Screen Shot 2020-09-25 at 11.32.51 AM.png|width=70%! > !Screen Shot 2020-09-25 at 11.35.01 AM.png|width=100%! > !Screen Shot 2020-09-25 at 11.36.48 AM.png|width=100%! > From above heap dump, Delta uses a SetAccumulator to records touched files > names > {code} > // Accumulator to collect all the distinct touched files > val touchedFilesAccum = new SetAccumulator[String]() > spark.sparkContext.register(touchedFilesAccum, TOUCHED_FILES_ACCUM_NAME) > // UDFs to records touched files names and add them to the accumulator > val recordTouchedFileName = udf { (fileName: String) => { > touchedFilesAccum.add(fileName) > 1 > }}.asNondeterministic() > {code} > In a big query, each task may hold thousands of file names, and if a stage > contains dozens of thousands of tasks, DAGscheduler may hold millions of > `CompletionEvent`. And each `CompletionEvent` holds the thousands of file > names in its `accumUpdates`. All accumulator objects will use Spark listener > event to deliver to the event loop and even a full GC can not release memory. > A PR will be submitted. With the patch, the memory problem was gone. > Before the patch: > !Screen Shot 2020-09-24 at 5.19.58 PM.png|width=70%! > After the patch: > !Screen Shot 2020-09-24 at 5.19.26 PM.png|width=70%! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem
[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Attachment: Screen Shot 2020-09-24 at 5.19.26 PM.png > External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may > lead driver full GC problem > - > > Key: SPARK-32994 > URL: https://issues.apache.org/jira/browse/SPARK-32994 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 2.4.7, 3.0.1, 3.1.0 >Reporter: Lantao Jin >Priority: Major > Attachments: Screen Shot 2020-09-24 at 5.19.26 PM.png, Screen Shot > 2020-09-24 at 5.19.58 PM.png, Screen Shot 2020-09-25 at 11.32.51 AM.png, > Screen Shot 2020-09-25 at 11.35.01 AM.png, Screen Shot 2020-09-25 at 11.36.48 > AM.png > > > We use Spark + Delta Lake, recently we find our Spark driver faced full GC > problem (very heavy) when users submit a MERGE INTO query. The driver held > over 100GB memory (depends on how much the max heap size set) and can not be > GC forever. By making a heap dump we found the root cause. > !Screen Shot 2020-09-25 at 11.32.51 AM.png|width=70%! > !Screen Shot 2020-09-25 at 11.35.01 AM.png|width=100%! > !Screen Shot 2020-09-25 at 11.36.48 AM.png|width=100%! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem
[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Attachment: Screen Shot 2020-09-24 at 5.19.58 PM.png > External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may > lead driver full GC problem > - > > Key: SPARK-32994 > URL: https://issues.apache.org/jira/browse/SPARK-32994 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 2.4.7, 3.0.1, 3.1.0 >Reporter: Lantao Jin >Priority: Major > Attachments: Screen Shot 2020-09-24 at 5.19.58 PM.png, Screen Shot > 2020-09-25 at 11.32.51 AM.png, Screen Shot 2020-09-25 at 11.35.01 AM.png, > Screen Shot 2020-09-25 at 11.36.48 AM.png > > > We use Spark + Delta Lake, recently we find our Spark driver faced full GC > problem (very heavy) when users submit a MERGE INTO query. The driver held > over 100GB memory (depends on how much the max heap size set) and can not be > GC forever. By making a heap dump we found the root cause. > !Screen Shot 2020-09-25 at 11.32.51 AM.png|width=70%! > !Screen Shot 2020-09-25 at 11.35.01 AM.png|width=100%! > !Screen Shot 2020-09-25 at 11.36.48 AM.png|width=100%! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem
[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Description: We use Spark + Delta Lake, recently we find our Spark driver faced full GC problem (very heavy) when users submit a MERGE INTO query. The driver held over 100GB memory (depends on how much the max heap size set) and can not be GC forever. By making a heap dump we found the root cause. !Screen Shot 2020-09-25 at 11.32.51 AM.png|width=70%! !Screen Shot 2020-09-25 at 11.35.01 AM.png|width=70%! !Screen Shot 2020-09-25 at 11.36.48 AM.png|width=70%! was: We use Spark + Delta Lake, recently we find our Spark driver faced full GC problem (very heavy) when users submit a MERGE INTO query. The driver held over 100GB memory (depends on how much the max heap size set) and can not be GC forever. By making a heap dump we found the root cause. !Screen Shot 2020-09-25 at 11.32.51 AM.png|width=100%! > External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may > lead driver full GC problem > - > > Key: SPARK-32994 > URL: https://issues.apache.org/jira/browse/SPARK-32994 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 2.4.7, 3.0.1, 3.1.0 >Reporter: Lantao Jin >Priority: Major > Attachments: Screen Shot 2020-09-25 at 11.32.51 AM.png, Screen Shot > 2020-09-25 at 11.35.01 AM.png, Screen Shot 2020-09-25 at 11.36.48 AM.png > > > We use Spark + Delta Lake, recently we find our Spark driver faced full GC > problem (very heavy) when users submit a MERGE INTO query. The driver held > over 100GB memory (depends on how much the max heap size set) and can not be > GC forever. By making a heap dump we found the root cause. > !Screen Shot 2020-09-25 at 11.32.51 AM.png|width=70%! > !Screen Shot 2020-09-25 at 11.35.01 AM.png|width=70%! > !Screen Shot 2020-09-25 at 11.36.48 AM.png|width=70%! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem
[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Description: We use Spark + Delta Lake, recently we find our Spark driver faced full GC problem (very heavy) when users submit a MERGE INTO query. The driver held over 100GB memory (depends on how much the max heap size set) and can not be GC forever. By making a heap dump we found the root cause. !Screen Shot 2020-09-25 at 11.32.51 AM.png|width=70%! !Screen Shot 2020-09-25 at 11.35.01 AM.png|width=100%! !Screen Shot 2020-09-25 at 11.36.48 AM.png|width=100%! was: We use Spark + Delta Lake, recently we find our Spark driver faced full GC problem (very heavy) when users submit a MERGE INTO query. The driver held over 100GB memory (depends on how much the max heap size set) and can not be GC forever. By making a heap dump we found the root cause. !Screen Shot 2020-09-25 at 11.32.51 AM.png|width=70%! !Screen Shot 2020-09-25 at 11.35.01 AM.png|width=70%! !Screen Shot 2020-09-25 at 11.36.48 AM.png|width=70%! > External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may > lead driver full GC problem > - > > Key: SPARK-32994 > URL: https://issues.apache.org/jira/browse/SPARK-32994 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 2.4.7, 3.0.1, 3.1.0 >Reporter: Lantao Jin >Priority: Major > Attachments: Screen Shot 2020-09-25 at 11.32.51 AM.png, Screen Shot > 2020-09-25 at 11.35.01 AM.png, Screen Shot 2020-09-25 at 11.36.48 AM.png > > > We use Spark + Delta Lake, recently we find our Spark driver faced full GC > problem (very heavy) when users submit a MERGE INTO query. The driver held > over 100GB memory (depends on how much the max heap size set) and can not be > GC forever. By making a heap dump we found the root cause. > !Screen Shot 2020-09-25 at 11.32.51 AM.png|width=70%! > !Screen Shot 2020-09-25 at 11.35.01 AM.png|width=100%! > !Screen Shot 2020-09-25 at 11.36.48 AM.png|width=100%! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem
[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Description: We use Spark + Delta Lake, recently we find our Spark driver faced full GC problem (very heavy) when users submit a MERGE INTO query. The driver held over 100GB memory (depends on how much the max heap size set) and can not be GC forever. By making a heap dump we found the root cause. !Screen Shot 2020-09-25 at 11.32.51 AM.png|width=100%! was: We use Spark + Delta Lake, recently we find our Spark driver faced full GC problem (very heavy) when users submit a MERGE INTO query. The driver held over 100GB memory (depends on how much the max heap size set) and can not be GC forever. By making a heap dump we found the root cause. !Screen Shot 2020-09-25 at 11.32.51 AM.png | width=100%! > External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may > lead driver full GC problem > - > > Key: SPARK-32994 > URL: https://issues.apache.org/jira/browse/SPARK-32994 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 2.4.7, 3.0.1, 3.1.0 >Reporter: Lantao Jin >Priority: Major > Attachments: Screen Shot 2020-09-25 at 11.32.51 AM.png, Screen Shot > 2020-09-25 at 11.35.01 AM.png, Screen Shot 2020-09-25 at 11.36.48 AM.png > > > We use Spark + Delta Lake, recently we find our Spark driver faced full GC > problem (very heavy) when users submit a MERGE INTO query. The driver held > over 100GB memory (depends on how much the max heap size set) and can not be > GC forever. By making a heap dump we found the root cause. > !Screen Shot 2020-09-25 at 11.32.51 AM.png|width=100%! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem
[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Description: We use Spark + Delta Lake, recently we find our Spark driver faced full GC problem (very heavy) when users submit a MERGE INTO query. The driver held over 100GB memory (depends on how much the max heap size set) and can not be GC forever. By making a heap dump we found the root cause. !Screen Shot 2020-09-25 at 11.32.51 AM.png! was: We use Spark + Delta Lake, recently we find our Spark driver faced full GC problem (very heavy) when users submit a MERGE INTO query. The driver held over 100GB memory (depends on how much the max heap size set) and can not be GC forever. By making a heap dump we found the root cause. > External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may > lead driver full GC problem > - > > Key: SPARK-32994 > URL: https://issues.apache.org/jira/browse/SPARK-32994 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 2.4.7, 3.0.1, 3.1.0 >Reporter: Lantao Jin >Priority: Major > Attachments: Screen Shot 2020-09-25 at 11.32.51 AM.png, Screen Shot > 2020-09-25 at 11.35.01 AM.png, Screen Shot 2020-09-25 at 11.36.48 AM.png > > > We use Spark + Delta Lake, recently we find our Spark driver faced full GC > problem (very heavy) when users submit a MERGE INTO query. The driver held > over 100GB memory (depends on how much the max heap size set) and can not be > GC forever. By making a heap dump we found the root cause. > !Screen Shot 2020-09-25 at 11.32.51 AM.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem
[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Attachment: Screen Shot 2020-09-25 at 11.36.48 AM.png > External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may > lead driver full GC problem > - > > Key: SPARK-32994 > URL: https://issues.apache.org/jira/browse/SPARK-32994 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 2.4.7, 3.0.1, 3.1.0 >Reporter: Lantao Jin >Priority: Major > Attachments: Screen Shot 2020-09-25 at 11.32.51 AM.png, Screen Shot > 2020-09-25 at 11.35.01 AM.png, Screen Shot 2020-09-25 at 11.36.48 AM.png > > > We use Spark + Delta Lake, recently we find our Spark driver faced full GC > problem (very heavy) when users submit a MERGE INTO query. The driver held > over 100GB memory (depends on how much the max heap size set) and can not be > GC forever. By making a heap dump we found the root cause. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem
[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Attachment: Screen Shot 2020-09-25 at 11.32.51 AM.png > External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may > lead driver full GC problem > - > > Key: SPARK-32994 > URL: https://issues.apache.org/jira/browse/SPARK-32994 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 2.4.7, 3.0.1, 3.1.0 >Reporter: Lantao Jin >Priority: Major > Attachments: Screen Shot 2020-09-25 at 11.32.51 AM.png, Screen Shot > 2020-09-25 at 11.35.01 AM.png, Screen Shot 2020-09-25 at 11.36.48 AM.png > > > We use Spark + Delta Lake, recently we find our Spark driver faced full GC > problem (very heavy) when users submit a MERGE INTO query. The driver held > over 100GB memory (depends on how much the max heap size set) and can not be > GC forever. By making a heap dump we found the root cause. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem
[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Attachment: Screen Shot 2020-09-25 at 11.35.01 AM.png > External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may > lead driver full GC problem > - > > Key: SPARK-32994 > URL: https://issues.apache.org/jira/browse/SPARK-32994 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 2.4.7, 3.0.1, 3.1.0 >Reporter: Lantao Jin >Priority: Major > Attachments: Screen Shot 2020-09-25 at 11.32.51 AM.png, Screen Shot > 2020-09-25 at 11.35.01 AM.png, Screen Shot 2020-09-25 at 11.36.48 AM.png > > > We use Spark + Delta Lake, recently we find our Spark driver faced full GC > problem (very heavy) when users submit a MERGE INTO query. The driver held > over 100GB memory (depends on how much the max heap size set) and can not be > GC forever. By making a heap dump we found the root cause. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem
[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Description: We use Spark + Delta Lake, recently we find our Spark driver faced full GC problem (very heavy) when users submit a MERGE INTO query. The driver held over 100GB memory (depends on how much the max heap size set) and can not be GC forever. By making a heap dump we found the root cause. !Screen Shot 2020-09-25 at 11.32.51 AM.png | width=100%! was: We use Spark + Delta Lake, recently we find our Spark driver faced full GC problem (very heavy) when users submit a MERGE INTO query. The driver held over 100GB memory (depends on how much the max heap size set) and can not be GC forever. By making a heap dump we found the root cause. !Screen Shot 2020-09-25 at 11.32.51 AM.png! > External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may > lead driver full GC problem > - > > Key: SPARK-32994 > URL: https://issues.apache.org/jira/browse/SPARK-32994 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 2.4.7, 3.0.1, 3.1.0 >Reporter: Lantao Jin >Priority: Major > Attachments: Screen Shot 2020-09-25 at 11.32.51 AM.png, Screen Shot > 2020-09-25 at 11.35.01 AM.png, Screen Shot 2020-09-25 at 11.36.48 AM.png > > > We use Spark + Delta Lake, recently we find our Spark driver faced full GC > problem (very heavy) when users submit a MERGE INTO query. The driver held > over 100GB memory (depends on how much the max heap size set) and can not be > GC forever. By making a heap dump we found the root cause. > !Screen Shot 2020-09-25 at 11.32.51 AM.png | width=100%! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org