Repository: spark
Updated Branches:
refs/heads/master 8e2169696 -> b60ee3a33
[SPARK-25307][SQL] ArraySort function may return an error in the code
generation phase
## What changes were proposed in this pull request?
Sorting array of booleans (not nullable) returns a compilation error in the
code generation phase. Below is the compilation error :
```SQL
java.util.concurrent.ExecutionException:
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 51,
Column 23: failed to compile: org.codehaus.commons.compiler.CompileException:
File 'generated.java', Line 51, Column 23: No applicable constructor/method
found for actual parameters "boolean[]"; candidates are: "public static void
java.util.Arrays.sort(long[])", "public static void
java.util.Arrays.sort(long[], int, int)", "public static void
java.util.Arrays.sort(byte[], int, int)", "public static void
java.util.Arrays.sort(float[])", "public static void
java.util.Arrays.sort(float[], int, int)", "public static void
java.util.Arrays.sort(char[])", "public static void
java.util.Arrays.sort(char[], int, int)", "public static void
java.util.Arrays.sort(short[], int, int)", "public static void
java.util.Arrays.sort(short[])", "public static void
java.util.Arrays.sort(byte[])", "public static void
java.util.Arrays.sort(java.lang.Object[], int, int, java.util.Comparator
)", "public static void java.util.Arrays.sort(java.lang.Object[],
java.util.Comparator)", "public static void java.util.Arrays.sort(int[])",
"public static void java.util.Arrays.sort(java.lang.Object[], int, int)",
"public static void java.util.Arrays.sort(java.lang.Object[])", "public static
void java.util.Arrays.sort(double[])", "public static void
java.util.Arrays.sort(double[], int, int)", "public static void
java.util.Arrays.sort(int[], int, int)"
at
com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:306)
at
com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:293)
at
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
at
com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:135)
at
com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2410)
at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2380)
at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2257)
at com.google.common.cache.LocalCache.get(LocalCache.java:4000)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4004)
at
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4874)
at
org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$.compile(CodeGenerator.scala:1305)
```
## How was this patch tested?
Added test in collectionExpressionSuite
Closes #22314 from dilipbiswal/SPARK-25307.
Authored-by: Dilip Biswal <[email protected]>
Signed-off-by: Takuya UESHIN <[email protected]>
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b60ee3a3
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b60ee3a3
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/b60ee3a3
Branch: refs/heads/master
Commit: b60ee3a337e0484e1d3e978197685d7d5ab858e7
Parents: 8e21696
Author: Dilip Biswal <[email protected]>
Authored: Tue Sep 4 13:39:29 2018 +0900
Committer: Takuya UESHIN <[email protected]>
Committed: Tue Sep 4 13:39:29 2018 +0900
----------------------------------------------------------------------
.../spark/sql/catalyst/expressions/collectionOperations.scala | 5 +++--
.../sql/catalyst/expressions/CollectionExpressionsSuite.scala | 7 +++++++
2 files changed, 10 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/b60ee3a3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
----------------------------------------------------------------------
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
index 17c683c..a29828e 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
@@ -1046,8 +1046,9 @@ trait ArraySortLike extends ExpectsInputTypes {
} else {
s"int $c = ${ctx.genComp(elementType, s"(($jt) $o1)", s"(($jt)
$o2)")};"
}
- val nonNullPrimitiveAscendingSort =
- if (CodeGenerator.isPrimitiveType(elementType) && !containsNull) {
+ val canPerformFastSort =
+ CodeGenerator.isPrimitiveType(elementType) && elementType !=
BooleanType && !containsNull
+ val nonNullPrimitiveAscendingSort = if (canPerformFastSort) {
val javaType = CodeGenerator.javaType(elementType)
val primitiveTypeName = CodeGenerator.primitiveTypeName(elementType)
s"""
http://git-wip-us.apache.org/repos/asf/spark/blob/b60ee3a3/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
----------------------------------------------------------------------
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
index a9fc3e9..96ae7d1 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
@@ -326,12 +326,19 @@ class CollectionExpressionsSuite extends SparkFunSuite
with ExpressionEvalHelper
val d2 = new Decimal().set(100)
val a4 = Literal.create(Seq(d2, d1), ArrayType(DecimalType(10, 0)))
val a5 = Literal.create(Seq(null, null), ArrayType(NullType))
+ val a6 = Literal.create(Seq(true, false, true, false),
+ ArrayType(BooleanType, containsNull = false))
+ val a7 = Literal.create(Seq(true, false, true, false),
ArrayType(BooleanType))
+ val a8 = Literal.create(Seq(true, false, true, null, false),
ArrayType(BooleanType))
checkEvaluation(new SortArray(a0), Seq(1, 2, 3))
checkEvaluation(new SortArray(a1), Seq[Integer]())
checkEvaluation(new SortArray(a2), Seq("a", "b"))
checkEvaluation(new SortArray(a3), Seq(null, "a", "b"))
checkEvaluation(new SortArray(a4), Seq(d1, d2))
+ checkEvaluation(new SortArray(a6), Seq(false, false, true, true))
+ checkEvaluation(new SortArray(a7), Seq(false, false, true, true))
+ checkEvaluation(new SortArray(a8), Seq(null, false, false, true, true))
checkEvaluation(SortArray(a0, Literal(true)), Seq(1, 2, 3))
checkEvaluation(SortArray(a1, Literal(true)), Seq[Integer]())
checkEvaluation(SortArray(a2, Literal(true)), Seq("a", "b"))
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
