FreakingMind opened a new issue, #7656:
URL: https://github.com/apache/datafusion-comet/issues/7656
### What is the problem the feature request solves?
When an INSERT has to narrow a numeric column (BIGINT into INT, DOUBLE into
BIGINT, DECIMAL(20,2) into DECIMAL(10,2)), Spark wraps the cast in
`CheckOverflowInTableInsert`. This happens under the default
`spark.sql.storeAssignmentPolicy=ANSI`, for V1 and DSv2 writes. Comet has no
serde for it, so the Project above the write falls back:
```
Execute InsertIntoHadoopFsRelationCommand
+- WriteFiles [COMET: Native support for operator WriteFilesExec is
disabled. ...]
+- Project [COMET: checkoverflowintableinsert is not supported]
+- CometColumnarToRow
+- CometNativeScan parquet spark_catalog.default.src
```
With `spark.comet.write.parquet.enabled=true` the write falls back too,
because its input is no longer Arrow. The plan string hides the wrapper, since
its `toString` prints only the child (`Project [cast(c#6L as int) AS c#15]`),
so the reason is easy to miss. We see this on Iceberg jobs that write into
narrower columns.
The wrapper only changes the error. If the cast throws a
`SparkArithmeticException`, Spark rethrows it as
`CAST_OVERFLOW_IN_TABLE_INSERT` with the column name and no value:
```
[CAST_OVERFLOW_IN_TABLE_INSERT] Fail to assign a value of "BIGINT" type to
the "INT" type column or variable `c` due to an overflow. ...
```
Passing the wrapper through would raise `CAST_OVERFLOW` with the value
instead, so it needs its own handling.
### Describe the potential solution
Run it natively: keep the cast native and turn an arithmetic error from it
into Spark's table-insert error.
### Additional context
Two details from Spark:
- The rename covers any `SparkArithmeticException` under the cast, not only
cast overflow. `INSERT INTO dst SELECT c div 0 FROM src` (BIGINT into INT)
fails with `CAST_OVERFLOW_IN_TABLE_INSERT`, not `DIVIDE_BY_ZERO`.
- When the child is no longer a `Cast` after optimization, Spark leaves the
error alone (SPARK-42286).
It is listed, unchecked, in #2314. The expression is the same in Spark 3.5
through 4.2; only the message text changed in 4.0.
### Willingness to contribute
I can contribute a fix for this bug independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]