philo-he opened a new issue, #13579:
URL: https://github.com/apache/gluten/issues/13579
### Description
### Background
Some Spark expression tests check that evaluation fails with a specific
exception type, for example:
```scala
val e = intercept[Exception] {
checkEvaluation(TruncTimestamp(Literal.create(fmt, StringType),
minTimestamp), null)
}.getCause
assert(e.isInstanceOf[ArithmeticException])
```
With Gluten, the native backend usually fails in the same case and for the
same reason, but the error reaches the driver as a different type (e.g.
`GlutenException` instead of `ArithmeticException`). Today such tests are
typically handled by excluding the Spark test and adding a modified copy as a
Gluten test. The copy has to be kept in sync with Spark's test across versions.
### Policy
Gluten aims for failure consistency with Spark: a query should fail when
Spark fails, for the same reason. The exception type the user sees is not
required to match. So a test whose only difference is the exception type should
run Spark's original test, with the type difference normalized, rather than be
excluded and copied.
The failure reason must still match. Normalization should only apply when
the native error clearly corresponds to Spark's (e.g. both are a `long
overflow`), so a native failure for an unrelated reason (unsupported type,
conversion error, ...) still fails the test.
### Approach
In `GlutenTestsTrait`, `doCheckExpression` is where an evaluation error
becomes the `fail(..., cause)` that Spark's tests inspect. A suite can override
it and replace the cause for a narrowly matched case, so Spark's test body runs
unchanged and the exclude can be removed.
Example from #13267 (`GlutenDateExpressionsSuite`, Spark 4.2):
```scala
// Velox raises the TruncTimestamp overflow with Spark's "long overflow"
message, but it reaches
// the driver as a GlutenException instead of an ArithmeticException.
Rethrow it as the
// ArithmeticException that Spark's test expects.
override protected def doCheckExpression(
expression: Expression,
expected: Any,
inputRow: InternalRow,
resultDF: DataFrame): Unit = {
try {
super.doCheckExpression(expression, expected, inputRow, resultDF)
} catch {
case e: TestFailedException
if expression.isInstanceOf[TruncTimestamp] &&
String.valueOf(e.getCause).contains("long overflow") =>
fail(e.getMessage, new ArithmeticException("long overflow"))
}
}
```
### Guidelines
- Match narrowly: the expression class plus a message that identifies the
failure reason. Don't map all native errors, or all "overflow" messages, to one
type.
- Add a short comment explaining which Spark exception is expected and why
the native error is equivalent.
- This is only needed for tests that catch the `checkEvaluation` failure and
inspect the cause. Tests using `checkExceptionInExpression[T]` already go
through `doCheckExceptionInExpression`, which searches the whole cause chain
and tolerates message differences for `SparkThrowable`.
- Keep copying a test only when the difference is more than the exception
type, e.g. a different result or a different failure condition.
### Possible follow-up
If more suites need this, `GlutenTestsTrait` could provide a hook so suites
only declare their mappings instead of repeating the try/catch:
```scala
/** Maps a native error to the exception Spark would throw, for an
equivalent failure. */
protected def toSparkException: PartialFunction[(Expression, Throwable),
Throwable] =
PartialFunction.empty
```
`doCheckExpression` would apply it to the cause before calling `fail`.
### Known limitation
This makes tests pass when the failure is consistent. It does not give
error-class parity: users can still see a different exception type, error class
or SQLSTATE than with vanilla Spark (e.g. through JDBC/Thrift clients, or
applications that check `getErrorClass`). Translating native errors into Spark
exceptions is separate work.
### References
- #13267: first use of this approach (`TruncTimestamp of Long.MinValue
overflows with ArithmeticException`, Spark 4.2)
### Gluten version
None
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]