GitHub user dongjoon-hyun opened a pull request:
https://github.com/apache/spark/pull/22150
[SPARK-25144][SQL][TEST][BRANCH-2.3] Free aggregate map when task ends
## What changes were proposed in this pull request?
[SPARK-25144](https://issues.apache.org/jira/browse/SPARK-25144) reports
memory leaks on Apache Spark 2.1.3, 2.2.2, 2.3.2-RC5.
```scala
scala> case class Foo(bar: Option[String])
scala> val ds = List(Foo(Some("bar"))).toDS
scala> val result = ds.flatMap(_.bar).distinct
scala> result.rdd.isEmpty
18/08/19 23:01:54 WARN Executor: Managed memory leak detected; size =
8650752 bytes, TID = 125
res0: Boolean = false
```
This is a backport of @cloud-fan 's
https://github.com/apache/spark/pull/21738 which is a single commit among 3
commits of SPARK-21743. In addition, I added a test case to prevent regressions
in branch-2.3 and branch-2.2. Although SPARK-21743 is reverted due to
regression, this subpatch can go to branch-2.3 and branch-2.2.
## How was this patch tested?
Pass the jenkins with a newly added test case.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dongjoon-hyun/spark SPARK-25144
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22150.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22150
----
commit 4355bf7f49b94dcfcfbe9df6be4c5c66cd0144e4
Author: Wenchen Fan <wenchen@...>
Date: 2018-07-10T15:07:10Z
[SPARK-25144][SQL] Free aggregate map when task ends
This is a backport of https://github.com/apache/spark/pull/21738.
Author: Wenchen Fan <[email protected]>
Closes #21738 from cloud-fan/limit.
commit 5f7f4466fb6192bc1b45d76e7c3a514d75e482ab
Author: Dongjoon Hyun <dongjoon@...>
Date: 2018-08-20T05:56:22Z
[SPARK-25144][SQL][TEST][BRANCH-2.3] Add a test case.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
