This is an automated email from the ASF dual-hosted git repository.
LuciferYang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 1e6bc6ea858b [SPARK-58506][SQL][TEST] Assert Spark's own error
contract in codegen compile failure tests
1e6bc6ea858b is described below
commit 1e6bc6ea858bec4a197f0f595eabcd458d80d9db
Author: YangJie <[email protected]>
AuthorDate: Mon Aug 3 17:08:53 2026 +0800
[SPARK-58506][SQL][TEST] Assert Spark's own error contract in codegen
compile failure tests
### What changes were proposed in this pull request?
Two tests around codegen compile failures assert on the compiler's own
diagnostic wording instead of what Spark itself produces. This PR points each
assertion at the part Spark owns.
`CodeGeneratorWithInterpretedFallbackSuite`, test "codegen failures in the
CODEGEN_ONLY mode":
```scala
assert(errMsg.contains("Failed to compile:
org.codehaus.commons.compiler.CompileException:"))
```
The class name in that string is Janino's. What Spark owns on this path is
the wrapping - a source-level failure goes through
`QueryExecutionErrors.compilerError`, whose checked `CompileException` the
compile cache wraps in an `ExecutionException`, which is exactly the contract
SPARK-23711 / SPARK-25140 established - and the `"Failed to compile: "` prefix
that `failedToCompileMsg` prepends. The test now asserts the cause's type and
that prefix. Asserting the cause type states the cont [...]
`ObjectExpressionsSuite`, test "SPARK-23593: InitializeJavaBean should
support interpreted execution":
```scala
checkExceptionInExpression[Exception](initializeWithNonexistingMethod,
"""A method named "nonexistent" is not declared in any enclosing class
""" +
"nor any supertype")
```
`checkExceptionInExpression` applies one substring to both the interpreted
and the codegen path, so this string has to match both. It does, but only
because the two happen to overlap. Interpreted execution raises Spark's own
`INTERNAL_ERROR` from `QueryExecutionErrors.methodNotDeclaredError`:
```
[INTERNAL_ERROR] A method named "nonexistent" is not declared in any
enclosing class nor any supertype SQLSTATE: XX000
```
while the codegen path fails in the Java compiler:
```
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line
41, Column 12: Failed to compile:
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 41,
Column 12: A method named "nonexistent" is not declared in any enclosing class
nor any supertype, nor through a static import
```
Janino's diagnostic is a superstring of Spark's sentence, so one assertion
covers both legs. The test now asserts each path against what raises it: the
interpreted leg through `checkError`, which pins the error class, parameters
and sqlState exactly - stronger than the substring it replaces - and the
codegen leg on the `ExecutionException` wrapping plus the `"Failed to compile:
"` prefix.
The same test also had an assertion that never ran, because the `contains`
result was discarded:
```scala
intercept[Exception] {
evaluateWithoutCodegen(initializeWithWrongParamType,
InternalRow.fromSeq(Seq()))
}.getMessage.contains(
"""A method named "setX" is not declared in any enclosing class """ +
"nor any supertype")
```
It has been a bare expression since SPARK-23593 introduced it in 2018, so
only "an exception was thrown" was ever verified. That leg raises the same
`methodNotDeclaredError`, so it now uses `checkError` as well rather than two
assertion styles for one error class in one test.
### Why are the changes needed?
An assertion on a compiler's diagnostic wording tests something Spark does
not control, and it obscures what the test is actually for. In
`CodeGeneratorWithInterpretedFallbackSuite` the contract under test is the
exception type, not the message. In `ObjectExpressionsSuite` the interpreted
path has an error class, parameters and a sqlState available, so a substring
match leaves assertable contract unasserted. And the `setX` assertion has
verified nothing for eight years.
Not addressed here, found while auditing for similar sites:
`DataFrameSuite`'s ignored test "SPARK-19372: Filter can be executed w/o
generated code due to JVM code size limit" asserts `e.contains("grows beyond 64
KiB")`, but Janino's string is `Code grows beyond 64 KB` - no released version
spells it `KiB`. That assertion is already stale and only survives because the
test is `ignore`d.
A side benefit: with the compiler's wording no longer asserted, these tests
no longer need updating if the generated code is ever compiled by something
other than Janino.
### Does this PR introduce _any_ user-facing change?
No. Test-only.
### How was this patch tested?
`CodeGeneratorWithInterpretedFallbackSuite`, `ObjectExpressionsSuite` and
`ExpressionEvalHelperSuite` (37 tests) pass, and `dev/scalastyle` is clean.
Each changed assertion was mutation-tested to confirm it is live: breaking
the expected message on the interpreted leg reports `checkError found 1
mismatch(es)`; breaking it on the `setX` leg does the same; changing the
codegen leg's `intercept` type to `TimeoutException` reports `Expected
exception java.util.concurrent.TimeoutException to be thrown, but
java.util.concurrent.ExecutionException was thrown`.
I also verified the claim that one substring covered both legs by accident:
the two real messages are quoted above, taken from a probe that printed them
under `CODEGEN_ONLY` and `NO_CODEGEN`. And to check that asserting the method
name alone would be too weak, I confirmed three unrelated failures still echo
it - wrapping the expression in an `Unevaluable`, switching the interpreted
path to `methodNotFoundError`, and falling back to a bare reflection exception
- because Spark echoes wh [...]
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code Opus 5
Closes #57707 from LuciferYang/SPARK-58506.
Authored-by: YangJie <[email protected]>
Signed-off-by: yangjie01 <[email protected]>
---
...CodeGeneratorWithInterpretedFallbackSuite.scala | 16 ++++++--
.../expressions/ObjectExpressionsSuite.scala | 47 ++++++++++++++++++----
2 files changed, 52 insertions(+), 11 deletions(-)
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CodeGeneratorWithInterpretedFallbackSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CodeGeneratorWithInterpretedFallbackSuite.scala
index a843d43ae83b..bf6351bba4b9 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CodeGeneratorWithInterpretedFallbackSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CodeGeneratorWithInterpretedFallbackSuite.scala
@@ -19,6 +19,8 @@ package org.apache.spark.sql.catalyst.expressions
import java.util.concurrent.ExecutionException
+import org.codehaus.commons.compiler.CompileException
+
import org.apache.spark.SparkFunSuite
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.expressions.aggregate.NoOp
@@ -83,13 +85,21 @@ class CodeGeneratorWithInterpretedFallbackSuite extends
SparkFunSuite with PlanT
}
test("codegen failures in the CODEGEN_ONLY mode") {
- val errMsg = intercept[ExecutionException] {
+ val e = intercept[ExecutionException] {
val input = Seq(BoundReference(0, IntegerType, nullable = true))
withSQLConf(SQLConf.CODEGEN_FACTORY_MODE.key -> codegenOnly) {
FailedCodegenProjection.createObject(input)
}
- }.getMessage
- assert(errMsg.contains("Failed to compile:
org.codehaus.commons.compiler.CompileException:"))
+ }
+ // SPARK-23711/SPARK-25140 made this path catch the exception the compile
cache raises rather
+ // than the compiler's own: a source-level failure goes through
`compilerError`, whose checked
+ // CompileException the cache wraps in an ExecutionException. (The other
branch,
+ // `internalCompilerError`, builds an unchecked InternalCompilerException,
which
+ // `CodeGenerator.compile` unwraps and rethrows bare.)
+ assert(e.getCause.isInstanceOf[CompileException])
+ // "Failed to compile: " comes from
`QueryExecutionErrors.failedToCompileMsg`; the compiler's
+ // own diagnostic follows it, and its wording is not Spark's to assert on.
+ assert(e.getMessage.contains("Failed to compile:"))
}
test("SPARK-25358 Correctly handles NoOp in MutableProjection") {
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ObjectExpressionsSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ObjectExpressionsSuite.scala
index 215362c47b94..f41f3e7df1b5 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ObjectExpressionsSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ObjectExpressionsSuite.scala
@@ -18,6 +18,7 @@
package org.apache.spark.sql.catalyst.expressions
import java.sql.{Date, Timestamp}
+import java.util.concurrent.ExecutionException
import scala.collection.immutable
import scala.collection.mutable
@@ -37,6 +38,7 @@ import
org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext, Genera
import org.apache.spark.sql.catalyst.expressions.objects._
import org.apache.spark.sql.catalyst.plans.logical.{LocalRelation, Project}
import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, ArrayData,
DateTimeUtils, GenericArrayData, IntervalUtils}
+import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types._
import org.apache.spark.unsafe.types.UTF8String
@@ -245,18 +247,47 @@ class ObjectExpressionsSuite extends SparkFunSuite with
ExpressionEvalHelper {
val initializeWithNonexistingMethod = InitializeJavaBean(
Literal.fromObject(new java.util.LinkedList[Int]),
Map("nonexistent" -> Literal(1)))
- checkExceptionInExpression[Exception](initializeWithNonexistingMethod,
- """A method named "nonexistent" is not declared in any enclosing class
""" +
- "nor any supertype")
+ // The two evaluation paths report this differently: interpreted execution
raises Spark's own
+ // INTERNAL_ERROR from `methodNotDeclaredError`, while codegen fails when
the Java compiler
+ // resolves the setter call in the generated source. Assert each on its
own rather than
+ // through one shared substring.
+ checkError(
+ exception = intercept[SparkException] {
+ evaluateWithoutCodegen(initializeWithNonexistingMethod,
InternalRow.fromSeq(Seq()))
+ },
+ condition = "INTERNAL_ERROR",
+ parameters = Map("message" ->
+ ("""A method named "nonexistent" is not declared in any enclosing
class """ +
+ "nor any supertype")),
+ sqlState = "XX000")
+ withSQLConf(
+ SQLConf.CODEGEN_FACTORY_MODE.key ->
CodegenObjectFactoryMode.CODEGEN_ONLY.toString) {
+ // What Spark owns on this path is the wrapping - an ExecutionException
around the
+ // CompileException that `compilerError` builds - and the "Failed to
compile: " prefix from
+ // `failedToCompileMsg`. The diagnostic after the prefix is the
compiler's own wording, so
+ // the setter name below is only a sanity check that it names the
offending member: the
+ // name is also the input, and Spark echoes whole expressions into other
errors, so on its
+ // own it would match unrelated failures too.
+ val errMsg = intercept[ExecutionException] {
+ evaluateWithMutableProjection(initializeWithNonexistingMethod,
InternalRow.fromSeq(Seq()))
+ }.getMessage
+ assert(errMsg.contains("Failed to compile:"))
+ assert(errMsg.contains("nonexistent"))
+ }
val initializeWithWrongParamType = InitializeJavaBean(
Literal.fromObject(new TestBean),
Map("setX" -> Literal("1")))
- intercept[Exception] {
- evaluateWithoutCodegen(initializeWithWrongParamType,
InternalRow.fromSeq(Seq()))
- }.getMessage.contains(
- """A method named "setX" is not declared in any enclosing class """ +
- "nor any supertype")
+ // Same error as above, reached only through interpreted execution.
+ checkError(
+ exception = intercept[SparkException] {
+ evaluateWithoutCodegen(initializeWithWrongParamType,
InternalRow.fromSeq(Seq()))
+ },
+ condition = "INTERNAL_ERROR",
+ parameters = Map("message" ->
+ ("""A method named "setX" is not declared in any enclosing class """ +
+ "nor any supertype")),
+ sqlState = "XX000")
}
test("InitializeJavaBean doesn't call setters if input in null") {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]