felipepessoto commented on issue #12779:
URL: https://github.com/apache/gluten/issues/12779#issuecomment-5301555806
## Updated root-cause analysis
Pulled the actual Delta v4.2.0 test sources to confirm the failure mechanism
precisely.
### `ImplicitMergeCastingSuite` / `ImplicitStreamingMergeCastingSuite`
From
`spark/src/test/scala/org/apache/spark/sql/delta/ImplicitDMLCastingSuite.scala`:
```scala
protected def expectLegacyCastingBehaviour(sqlConfig: SqlConfiguration):
Boolean = {
(sqlConfig.followAnsiEnabled && !sqlConfig.ansiEnabled) ||
(!sqlConfig.followAnsiEnabled && sqlConfig.storeAssignmentPolicy ==
SQLConf.StoreAssignmentPolicy.LEGACY)
}
...
if (expectLegacyCastingBehaviour(sqlConfig)) {
sql(mergeCommand) // no exception expected -- legacy/silent truncation
} else {
val exception = intercept[Throwable] { sql(mergeCommand) }
validateException(exception, sqlConfig, testConfig)
}
```
For the failing config (`followAnsiEnabled: false, ansiEnabled: true,
storeAssignmentPolicy: LEGACY`), `expectLegacyCastingBehaviour` returns
**true**. Delta intentionally decouples its own MERGE/UPDATE cast strictness
(`DELTA_UPDATE_AND_MERGE_CASTING_FOLLOWS_ANSI_ENABLED_FLAG`) from the global
`spark.sql.ansi.enabled` flag when that flag is `false` -- in that case
`storeAssignmentPolicy` alone governs, and `LEGACY` means the overflowing
decimal cast should be **silently truncated**, not throw.
### `TypeWideningInsertSchemaEvolutionBasicSuite#INSERT - never automatic
type widening`
Similarly, this test explicitly sets `SQLConf.STORE_ASSIGNMENT_POLICY.key ->
LEGACY` and simply performs the write with no `intercept` block at all -- it
only asserts the write succeeds and the schema is *not* widened.
### Conclusion
**These tests do not expect any exception.** The regression is that
Gluten/Velox now throws where vanilla Spark would silently truncate/overflow
under `StoreAssignmentPolicy.LEGACY`.
Most likely explanation: Gluten forwards Spark's *global*
`spark.sql.ansi.enabled` setting to Velox's native DECIMAL-to-DECIMAL cast
(which, after upstream Velox commit `a96d0415c` "fix(spark): Enable
ANSI-compliant cast from DECIMAL to DECIMAL", now enforces ANSI
overflow-checking based on that flag). It does not appear to account for
Delta's separate, more granular `storeAssignmentPolicy` /
`updateAndMergeCastingFollowsAnsiEnabled` logic that determines whether a
specific MERGE/UPDATE/INSERT assignment cast should be strict or legacy
independent of the global ANSI flag. So when `ansiEnabled=true` globally but
Delta wants legacy (non-strict) semantics for this particular assignment,
Velox's native cast throws anyway.
### Suggested next step
Investigate how Gluten maps decimal cast overflow/ANSI behavior for
table-write assignment casts (MERGE/UPDATE/INSERT `mergeSchema` paths) -- it
likely needs to honor `spark.sql.storeAssignmentPolicy` (and Delta's
cast-policy override) rather than only `spark.sql.ansi.enabled` when deciding
whether Velox's decimal cast should raise on overflow.
Generated-by: Claude claude-sonnet-5 (GitHub Copilot CLI)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
