This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 931e68b88d0b [SPARK-46139][SQL][TESTS] Fix `QueryExecutionErrorsSuite`
with Java 21
931e68b88d0b is described below
commit 931e68b88d0b34d81eca48c1650ad9df95c75c15
Author: yangjie01 <yangji...@baidu.com>
AuthorDate: Tue Nov 28 08:10:13 2023 -0800
[SPARK-46139][SQL][TESTS] Fix `QueryExecutionErrorsSuite` with Java 21
### What changes were proposed in this pull request?
In the Java 21 daily tests, there are two test failures in
`QueryExecutionErrorsSuite`: execute user defined function with registered UDF`
and `FAILED_EXECUTE_UDF: execute user defined function`
- https://github.com/apache/spark/actions/runs/6993283399/job/19066715935
- https://github.com/apache/spark/actions/runs/7000424685/job/19066731946
- https://github.com/apache/spark/actions/runs/7014008773/job/19081075487
```
[info] - FAILED_EXECUTE_UDF: execute user defined function with registered
UDF *** FAILED *** (48 milliseconds)
[info] java.lang.IllegalArgumentException: For parameter 'reason' value
'java.lang.StringIndexOutOfBoundsException: Range [5, 6) out of bounds for
length 5' does not match: java.lang.StringIndexOutOfBoundsException: begin 5,
end 6, length 5
[info] at
org.apache.spark.SparkFunSuite.$anonfun$checkError$2(SparkFunSuite.scala:357)
[info] at
org.apache.spark.SparkFunSuite.$anonfun$checkError$2$adapted(SparkFunSuite.scala:352)
[info] at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:576)
[info] at
scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:574)
[info] at scala.collection.AbstractIterable.foreach(Iterable.scala:933)
[info] at
org.apache.spark.SparkFunSuite.checkError(SparkFunSuite.scala:352)
[info] at
org.apache.spark.sql.errors.QueryExecutionErrorsSuite.$anonfun$new$58(QueryExecutionErrorsSuite.scala:428)
...
[info] - FAILED_EXECUTE_UDF: execute user defined function *** FAILED ***
(32 milliseconds)
[info] java.lang.IllegalArgumentException: For parameter 'reason' value
'java.lang.StringIndexOutOfBoundsException: Range [5, 6) out of bounds for
length 5' does not match: java.lang.StringIndexOutOfBoundsException: begin 5,
end 6, length 5
[info] at
org.apache.spark.SparkFunSuite.$anonfun$checkError$2(SparkFunSuite.scala:357)
[info] at
org.apache.spark.SparkFunSuite.$anonfun$checkError$2$adapted(SparkFunSuite.scala:352)
[info] at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:576)
[info] at
scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:574)
[info] at scala.collection.AbstractIterable.foreach(Iterable.scala:933)
[info] at
org.apache.spark.SparkFunSuite.checkError(SparkFunSuite.scala:352)
[info] at
org.apache.spark.sql.errors.QueryExecutionErrorsSuite.$anonfun$new$61(QueryExecutionErrorsSuite.scala:454)
```
This is due to different error messages when testing with Java 21 compared
to Java 17:
- Java 17
```
org.apache.spark.SparkException: [FAILED_EXECUTE_UDF] User defined function
(`luckyCharOfWord (QueryExecutionErrorsSuite$$Lambda$4610/0x000000e002355808)`:
(string, int) => string) failed due to:
java.lang.StringIndexOutOfBoundsException: begin 5, end 6, length 5. SQLSTATE:
39000
at
org.apache.spark.sql.errors.QueryExecutionErrors$.failedExecuteUserDefinedFunctionError(QueryExecutionErrors.scala:195)
at
org.apache.spark.sql.errors.QueryExecutionErrors.failedExecuteUserDefinedFunctionError(QueryExecutionErrors.scala)
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown
Source)
at
org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at
org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:43)
at
org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:388)
at
org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:891)
```
- Java 21
```
org.apache.spark.SparkException: [FAILED_EXECUTE_UDF] User defined function
(`luckyCharOfWord (QueryExecutionErrorsSuite$$Lambda/0x00000070022eec08)`:
(string, int) => string) failed due to:
java.lang.StringIndexOutOfBoundsException: Range [5, 6) out of bounds for
length 5. SQLSTATE: 39000
at
org.apache.spark.sql.errors.QueryExecutionErrors$.failedExecuteUserDefinedFunctionError(QueryExecutionErrors.scala:195)
at
org.apache.spark.sql.errors.QueryExecutionErrors.failedExecuteUserDefinedFunctionError(QueryExecutionErrors.scala)
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown
Source)
at
org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at
org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:43)
at
org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:388)
at
org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:891)
```
This PR makes the tests pass by using different `reason` for matching based
on the different Java versions.
### Why are the changes needed?
Fix `QueryExecutionErrorsSuite` with Java 21
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Run `build/sbt "sql/testOnly
org.apache.spark.sql.errors.QueryExecutionErrorsSuite"` with Java 21, all test
passed.
```
java -version
openjdk version "21.0.1" 2023-10-17 LTS
OpenJDK Runtime Environment Zulu21.30+15-CA (build 21.0.1+12-LTS)
OpenJDK 64-Bit Server VM Zulu21.30+15-CA (build 21.0.1+12-LTS, mixed mode,
sharing)
build/sbt "sql/testOnly
org.apache.spark.sql.errors.QueryExecutionErrorsSuite"
[info] Run completed in 8 seconds, 951 milliseconds.
[info] Total number of tests run: 49
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 49, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
```
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #44056 from LuciferYang/SPARK-46139.
Authored-by: yangjie01 <yangji...@baidu.com>
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
.../spark/sql/errors/QueryExecutionErrorsSuite.scala | 14 ++++++++++++--
1 file changed, 12 insertions(+), 2 deletions(-)
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/errors/QueryExecutionErrorsSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/errors/QueryExecutionErrorsSuite.scala
index 1e869bfd25aa..e70c3d62b34d 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/errors/QueryExecutionErrorsSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/errors/QueryExecutionErrorsSuite.scala
@@ -424,6 +424,11 @@ class QueryExecutionErrorsSuite
} else {
"`luckyCharOfWord
\\(QueryExecutionErrorsSuite\\$\\$Lambda\\$\\d+/\\w+\\)`"
}
+ val reason = if (Utils.isJavaVersionAtLeast21) {
+ "java.lang.StringIndexOutOfBoundsException: Range \\[5, 6\\) out of
bounds for length 5"
+ } else {
+ "java.lang.StringIndexOutOfBoundsException: begin 5, end 6, length 5"
+ }
checkError(
exception = e.getCause.asInstanceOf[SparkException],
@@ -432,7 +437,7 @@ class QueryExecutionErrorsSuite
"functionName" -> functionNameRegex,
"signature" -> "string, int",
"result" -> "string",
- "reason" -> "java.lang.StringIndexOutOfBoundsException: begin 5, end
6, length 5"),
+ "reason" -> reason),
matchPVals = true)
}
@@ -450,6 +455,11 @@ class QueryExecutionErrorsSuite
} else {
"`QueryExecutionErrorsSuite\\$\\$Lambda\\$\\d+/\\w+`"
}
+ val reason = if (Utils.isJavaVersionAtLeast21) {
+ "java.lang.StringIndexOutOfBoundsException: Range \\[5, 6\\) out of
bounds for length 5"
+ } else {
+ "java.lang.StringIndexOutOfBoundsException: begin 5, end 6, length 5"
+ }
checkError(
exception = e.getCause.asInstanceOf[SparkException],
@@ -457,7 +467,7 @@ class QueryExecutionErrorsSuite
parameters = Map("functionName" -> functionNameRegex,
"signature" -> "string, int",
"result" -> "string",
- "reason" -> "java.lang.StringIndexOutOfBoundsException: begin 5, end
6, length 5"),
+ "reason" -> reason),
matchPVals = true)
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
