This is an automated email from the ASF dual-hosted git repository.

LuciferYang pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 0c5a815817c5 [SPARK-58506][SQL][TEST] Assert Spark's own error 
contract in codegen compile failure tests
0c5a815817c5 is described below

commit 0c5a815817c5e997002b15825612636405ad5e78
Author: YangJie <[email protected]>
AuthorDate: Mon Aug 3 17:08:53 2026 +0800

    [SPARK-58506][SQL][TEST] Assert Spark's own error contract in codegen 
compile failure tests
    
    ### What changes were proposed in this pull request?
    
    Two tests around codegen compile failures assert on the compiler's own 
diagnostic wording instead of what Spark itself produces. This PR points each 
assertion at the part Spark owns.
    
    `CodeGeneratorWithInterpretedFallbackSuite`, test "codegen failures in the 
CODEGEN_ONLY mode":
    
    ```scala
    assert(errMsg.contains("Failed to compile: 
org.codehaus.commons.compiler.CompileException:"))
    ```
    
    The class name in that string is Janino's. What Spark owns on this path is 
the wrapping - a source-level failure goes through 
`QueryExecutionErrors.compilerError`, whose checked `CompileException` the 
compile cache wraps in an `ExecutionException`, which is exactly the contract 
SPARK-23711 / SPARK-25140 established - and the `"Failed to compile: "` prefix 
that `failedToCompileMsg` prepends. The test now asserts the cause's type and 
that prefix. Asserting the cause type states the cont [...]
    
    `ObjectExpressionsSuite`, test "SPARK-23593: InitializeJavaBean should 
support interpreted execution":
    
    ```scala
    checkExceptionInExpression[Exception](initializeWithNonexistingMethod,
      """A method named "nonexistent" is not declared in any enclosing class 
""" +
        "nor any supertype")
    ```
    
    `checkExceptionInExpression` applies one substring to both the interpreted 
and the codegen path, so this string has to match both. It does, but only 
because the two happen to overlap. Interpreted execution raises Spark's own 
`INTERNAL_ERROR` from `QueryExecutionErrors.methodNotDeclaredError`:
    
    ```
    [INTERNAL_ERROR] A method named "nonexistent" is not declared in any 
enclosing class nor any supertype SQLSTATE: XX000
    ```
    
    while the codegen path fails in the Java compiler:
    
    ```
    org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 
41, Column 12: Failed to compile: 
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 41, 
Column 12: A method named "nonexistent" is not declared in any enclosing class 
nor any supertype, nor through a static import
    ```
    
    Janino's diagnostic is a superstring of Spark's sentence, so one assertion 
covers both legs. The test now asserts each path against what raises it: the 
interpreted leg through `checkError`, which pins the error class, parameters 
and sqlState exactly - stronger than the substring it replaces - and the 
codegen leg on the `ExecutionException` wrapping plus the `"Failed to compile: 
"` prefix.
    
    The same test also had an assertion that never ran, because the `contains` 
result was discarded:
    
    ```scala
    intercept[Exception] {
      evaluateWithoutCodegen(initializeWithWrongParamType, 
InternalRow.fromSeq(Seq()))
    }.getMessage.contains(
      """A method named "setX" is not declared in any enclosing class """ +
        "nor any supertype")
    ```
    
    It has been a bare expression since SPARK-23593 introduced it in 2018, so 
only "an exception was thrown" was ever verified. That leg raises the same 
`methodNotDeclaredError`, so it now uses `checkError` as well rather than two 
assertion styles for one error class in one test.
    
    ### Why are the changes needed?
    
    An assertion on a compiler's diagnostic wording tests something Spark does 
not control, and it obscures what the test is actually for. In 
`CodeGeneratorWithInterpretedFallbackSuite` the contract under test is the 
exception type, not the message. In `ObjectExpressionsSuite` the interpreted 
path has an error class, parameters and a sqlState available, so a substring 
match leaves assertable contract unasserted. And the `setX` assertion has 
verified nothing for eight years.
    
    Not addressed here, found while auditing for similar sites: 
`DataFrameSuite`'s ignored test "SPARK-19372: Filter can be executed w/o 
generated code due to JVM code size limit" asserts `e.contains("grows beyond 64 
KiB")`, but Janino's string is `Code grows beyond 64 KB` - no released version 
spells it `KiB`. That assertion is already stale and only survives because the 
test is `ignore`d.
    
    A side benefit: with the compiler's wording no longer asserted, these tests 
no longer need updating if the generated code is ever compiled by something 
other than Janino.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. Test-only.
    
    ### How was this patch tested?
    
    `CodeGeneratorWithInterpretedFallbackSuite`, `ObjectExpressionsSuite` and 
`ExpressionEvalHelperSuite` (37 tests) pass, and `dev/scalastyle` is clean.
    
    Each changed assertion was mutation-tested to confirm it is live: breaking 
the expected message on the interpreted leg reports `checkError found 1 
mismatch(es)`; breaking it on the `setX` leg does the same; changing the 
codegen leg's `intercept` type to `TimeoutException` reports `Expected 
exception java.util.concurrent.TimeoutException to be thrown, but 
java.util.concurrent.ExecutionException was thrown`.
    
    I also verified the claim that one substring covered both legs by accident: 
the two real messages are quoted above, taken from a probe that printed them 
under `CODEGEN_ONLY` and `NO_CODEGEN`. And to check that asserting the method 
name alone would be too weak, I confirmed three unrelated failures still echo 
it - wrapping the expression in an `Unevaluable`, switching the interpreted 
path to `methodNotFoundError`, and falling back to a bare reflection exception 
- because Spark echoes wh [...]
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code Opus 5
    
    Closes #57707 from LuciferYang/SPARK-58506.
    
    Authored-by: YangJie <[email protected]>
    Signed-off-by: yangjie01 <[email protected]>
    (cherry picked from commit 1e6bc6ea858bec4a197f0f595eabcd458d80d9db)
    Signed-off-by: yangjie01 <[email protected]>
---
 ...CodeGeneratorWithInterpretedFallbackSuite.scala | 16 ++++++--
 .../expressions/ObjectExpressionsSuite.scala       | 47 ++++++++++++++++++----
 2 files changed, 52 insertions(+), 11 deletions(-)

diff --git 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CodeGeneratorWithInterpretedFallbackSuite.scala
 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CodeGeneratorWithInterpretedFallbackSuite.scala
index a843d43ae83b..bf6351bba4b9 100644
--- 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CodeGeneratorWithInterpretedFallbackSuite.scala
+++ 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CodeGeneratorWithInterpretedFallbackSuite.scala
@@ -19,6 +19,8 @@ package org.apache.spark.sql.catalyst.expressions
 
 import java.util.concurrent.ExecutionException
 
+import org.codehaus.commons.compiler.CompileException
+
 import org.apache.spark.SparkFunSuite
 import org.apache.spark.sql.catalyst.InternalRow
 import org.apache.spark.sql.catalyst.expressions.aggregate.NoOp
@@ -83,13 +85,21 @@ class CodeGeneratorWithInterpretedFallbackSuite extends 
SparkFunSuite with PlanT
   }
 
   test("codegen failures in the CODEGEN_ONLY mode") {
-    val errMsg = intercept[ExecutionException] {
+    val e = intercept[ExecutionException] {
       val input = Seq(BoundReference(0, IntegerType, nullable = true))
       withSQLConf(SQLConf.CODEGEN_FACTORY_MODE.key -> codegenOnly) {
         FailedCodegenProjection.createObject(input)
       }
-    }.getMessage
-    assert(errMsg.contains("Failed to compile: 
org.codehaus.commons.compiler.CompileException:"))
+    }
+    // SPARK-23711/SPARK-25140 made this path catch the exception the compile 
cache raises rather
+    // than the compiler's own: a source-level failure goes through 
`compilerError`, whose checked
+    // CompileException the cache wraps in an ExecutionException. (The other 
branch,
+    // `internalCompilerError`, builds an unchecked InternalCompilerException, 
which
+    // `CodeGenerator.compile` unwraps and rethrows bare.)
+    assert(e.getCause.isInstanceOf[CompileException])
+    // "Failed to compile: " comes from 
`QueryExecutionErrors.failedToCompileMsg`; the compiler's
+    // own diagnostic follows it, and its wording is not Spark's to assert on.
+    assert(e.getMessage.contains("Failed to compile:"))
   }
 
   test("SPARK-25358 Correctly handles NoOp in MutableProjection") {
diff --git 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ObjectExpressionsSuite.scala
 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ObjectExpressionsSuite.scala
index 215362c47b94..f41f3e7df1b5 100644
--- 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ObjectExpressionsSuite.scala
+++ 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ObjectExpressionsSuite.scala
@@ -18,6 +18,7 @@
 package org.apache.spark.sql.catalyst.expressions
 
 import java.sql.{Date, Timestamp}
+import java.util.concurrent.ExecutionException
 
 import scala.collection.immutable
 import scala.collection.mutable
@@ -37,6 +38,7 @@ import 
org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext, Genera
 import org.apache.spark.sql.catalyst.expressions.objects._
 import org.apache.spark.sql.catalyst.plans.logical.{LocalRelation, Project}
 import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, ArrayData, 
DateTimeUtils, GenericArrayData, IntervalUtils}
+import org.apache.spark.sql.internal.SQLConf
 import org.apache.spark.sql.types._
 import org.apache.spark.unsafe.types.UTF8String
 
@@ -245,18 +247,47 @@ class ObjectExpressionsSuite extends SparkFunSuite with 
ExpressionEvalHelper {
     val initializeWithNonexistingMethod = InitializeJavaBean(
       Literal.fromObject(new java.util.LinkedList[Int]),
       Map("nonexistent" -> Literal(1)))
-    checkExceptionInExpression[Exception](initializeWithNonexistingMethod,
-      """A method named "nonexistent" is not declared in any enclosing class 
""" +
-        "nor any supertype")
+    // The two evaluation paths report this differently: interpreted execution 
raises Spark's own
+    // INTERNAL_ERROR from `methodNotDeclaredError`, while codegen fails when 
the Java compiler
+    // resolves the setter call in the generated source. Assert each on its 
own rather than
+    // through one shared substring.
+    checkError(
+      exception = intercept[SparkException] {
+        evaluateWithoutCodegen(initializeWithNonexistingMethod, 
InternalRow.fromSeq(Seq()))
+      },
+      condition = "INTERNAL_ERROR",
+      parameters = Map("message" ->
+        ("""A method named "nonexistent" is not declared in any enclosing 
class """ +
+          "nor any supertype")),
+      sqlState = "XX000")
+    withSQLConf(
+        SQLConf.CODEGEN_FACTORY_MODE.key -> 
CodegenObjectFactoryMode.CODEGEN_ONLY.toString) {
+      // What Spark owns on this path is the wrapping - an ExecutionException 
around the
+      // CompileException that `compilerError` builds - and the "Failed to 
compile: " prefix from
+      // `failedToCompileMsg`. The diagnostic after the prefix is the 
compiler's own wording, so
+      // the setter name below is only a sanity check that it names the 
offending member: the
+      // name is also the input, and Spark echoes whole expressions into other 
errors, so on its
+      // own it would match unrelated failures too.
+      val errMsg = intercept[ExecutionException] {
+        evaluateWithMutableProjection(initializeWithNonexistingMethod, 
InternalRow.fromSeq(Seq()))
+      }.getMessage
+      assert(errMsg.contains("Failed to compile:"))
+      assert(errMsg.contains("nonexistent"))
+    }
 
     val initializeWithWrongParamType = InitializeJavaBean(
       Literal.fromObject(new TestBean),
       Map("setX" -> Literal("1")))
-    intercept[Exception] {
-      evaluateWithoutCodegen(initializeWithWrongParamType, 
InternalRow.fromSeq(Seq()))
-    }.getMessage.contains(
-      """A method named "setX" is not declared in any enclosing class """ +
-        "nor any supertype")
+    // Same error as above, reached only through interpreted execution.
+    checkError(
+      exception = intercept[SparkException] {
+        evaluateWithoutCodegen(initializeWithWrongParamType, 
InternalRow.fromSeq(Seq()))
+      },
+      condition = "INTERNAL_ERROR",
+      parameters = Map("message" ->
+        ("""A method named "setX" is not declared in any enclosing class """ +
+          "nor any supertype")),
+      sqlState = "XX000")
   }
 
   test("InitializeJavaBean doesn't call setters if input in null") {


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to