spark git commit: [SPARK-12690][CORE] Fix NPE in UnsafeInMemorySorter.free()

[joshrosen]

Repository: spark
Updated Branches:
  refs/heads/branch-1.6 cf6d506c7 -> 26f13faa9


[SPARK-12690][CORE] Fix NPE in UnsafeInMemorySorter.free()

I hit the exception below. The `UnsafeKVExternalSorter` does pass `null` as the 
consumer when creating an `UnsafeInMemorySorter`. Normally the NPE doesn't 
occur because the `inMemSorter` is set to null later and the `free()` method is 
not called. It happens when there is another exception like OOM thrown before 
setting `inMemSorter` to null. Anyway, we can add the null check to avoid it.

```
ERROR spark.TaskContextImpl: Error in TaskCompletionListener
java.lang.NullPointerException
at 
org.apache.spark.util.collection.unsafe.sort.UnsafeInMemorySorter.free(UnsafeInMemorySorter.java:110)
at 
org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.cleanupResources(UnsafeExternalSorter.java:288)
at 
org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter$1.onTaskCompletion(UnsafeExternalSorter.java:141)
at 
org.apache.spark.TaskContextImpl$$anonfun$markTaskCompleted$1.apply(TaskContextImpl.scala:79)
at 
org.apache.spark.TaskContextImpl$$anonfun$markTaskCompleted$1.apply(TaskContextImpl.scala:77)
at 
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at 
org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:77)
at org.apache.spark.scheduler.Task.run(Task.scala:91)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
```

Author: Carson Wang 

Closes #10637 from carsonwang/FixNPE.

(cherry picked from commit eabc7b8ee7e809bab05361ed154f87bff467bd88)
Signed-off-by: Josh Rosen 


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/26f13faa
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/26f13faa
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/26f13faa

Branch: refs/heads/branch-1.6
Commit: 26f13faa981a51046ed1f16b9c3ee42ac5f6b6da
Parents: cf6d506
Author: Carson Wang 
Authored: Wed Jan 13 13:28:39 2016 -0800
Committer: Josh Rosen 
Committed: Wed Jan 13 13:29:18 2016 -0800

--
 .../util/collection/unsafe/sort/UnsafeInMemorySorter.java  | 6 --
 1 file changed, 4 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/26f13faa/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java
--
diff --git 
a/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java
 
b/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java
index dce1f15..98a7314 100644
--- 
a/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java
+++ 
b/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java
@@ -107,8 +107,10 @@ public final class UnsafeInMemorySorter {
* Free the memory used by pointer array.
*/
   public void free() {
-consumer.freeArray(array);
-array = null;
+if (consumer != null) {
+  consumer.freeArray(array);
+  array = null;
+}
   }
 
   public void reset() {


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

spark git commit: [SPARK-12690][CORE] Fix NPE in UnsafeInMemorySorter.free()

[joshrosen]

Repository: spark
Updated Branches:
  refs/heads/master cbbcd8e42 -> eabc7b8ee


[SPARK-12690][CORE] Fix NPE in UnsafeInMemorySorter.free()

I hit the exception below. The `UnsafeKVExternalSorter` does pass `null` as the 
consumer when creating an `UnsafeInMemorySorter`. Normally the NPE doesn't 
occur because the `inMemSorter` is set to null later and the `free()` method is 
not called. It happens when there is another exception like OOM thrown before 
setting `inMemSorter` to null. Anyway, we can add the null check to avoid it.

```
ERROR spark.TaskContextImpl: Error in TaskCompletionListener
java.lang.NullPointerException
at 
org.apache.spark.util.collection.unsafe.sort.UnsafeInMemorySorter.free(UnsafeInMemorySorter.java:110)
at 
org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.cleanupResources(UnsafeExternalSorter.java:288)
at 
org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter$1.onTaskCompletion(UnsafeExternalSorter.java:141)
at 
org.apache.spark.TaskContextImpl$$anonfun$markTaskCompleted$1.apply(TaskContextImpl.scala:79)
at 
org.apache.spark.TaskContextImpl$$anonfun$markTaskCompleted$1.apply(TaskContextImpl.scala:77)
at 
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at 
org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:77)
at org.apache.spark.scheduler.Task.run(Task.scala:91)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
```

Author: Carson Wang 

Closes #10637 from carsonwang/FixNPE.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/eabc7b8e
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/eabc7b8e
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/eabc7b8e

Branch: refs/heads/master
Commit: eabc7b8ee7e809bab05361ed154f87bff467bd88
Parents: cbbcd8e
Author: Carson Wang 
Authored: Wed Jan 13 13:28:39 2016 -0800
Committer: Josh Rosen 
Committed: Wed Jan 13 13:28:39 2016 -0800

--
 .../util/collection/unsafe/sort/UnsafeInMemorySorter.java  | 6 --
 1 file changed, 4 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/eabc7b8e/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java
--
diff --git 
a/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java
 
b/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java
index f71b8d1..d1b0bc5 100644
--- 
a/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java
+++ 
b/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java
@@ -116,8 +116,10 @@ public final class UnsafeInMemorySorter {
* Free the memory used by pointer array.
*/
   public void free() {
-consumer.freeArray(array);
-array = null;
+if (consumer != null) {
+  consumer.freeArray(array);
+  array = null;
+}
   }
 
   public void reset() {


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org