FreakingMind opened a new issue, #7656:
URL: https://github.com/apache/datafusion-comet/issues/7656

   ### What is the problem the feature request solves?
   
   When an INSERT has to narrow a numeric column (BIGINT into INT, DOUBLE into 
BIGINT, DECIMAL(20,2) into DECIMAL(10,2)), Spark wraps the cast in 
`CheckOverflowInTableInsert`. This happens under the default 
`spark.sql.storeAssignmentPolicy=ANSI`, for V1 and DSv2 writes. Comet has no 
serde for it, so the Project above the write falls back:
   
   ```
   Execute InsertIntoHadoopFsRelationCommand
   +-  WriteFiles [COMET: Native support for operator WriteFilesExec is 
disabled. ...]
      +-  Project [COMET: checkoverflowintableinsert is not supported]
         +- CometColumnarToRow
            +- CometNativeScan parquet spark_catalog.default.src
   ```
   
   With `spark.comet.write.parquet.enabled=true` the write falls back too, 
because its input is no longer Arrow. The plan string hides the wrapper, since 
its `toString` prints only the child (`Project [cast(c#6L as int) AS c#15]`), 
so the reason is easy to miss. We see this on Iceberg jobs that write into 
narrower columns.
   
   The wrapper only changes the error. If the cast throws a 
`SparkArithmeticException`, Spark rethrows it as 
`CAST_OVERFLOW_IN_TABLE_INSERT` with the column name and no value:
   
   ```
   [CAST_OVERFLOW_IN_TABLE_INSERT] Fail to assign a value of "BIGINT" type to 
the "INT" type column or variable `c` due to an overflow. ...
   ```
   
   Passing the wrapper through would raise `CAST_OVERFLOW` with the value 
instead, so it needs its own handling.
   
   ### Describe the potential solution
   
   Run it natively: keep the cast native and turn an arithmetic error from it 
into Spark's table-insert error.
   
   ### Additional context
   
   Two details from Spark:
   
   - The rename covers any `SparkArithmeticException` under the cast, not only 
cast overflow. `INSERT INTO dst SELECT c div 0 FROM src` (BIGINT into INT) 
fails with `CAST_OVERFLOW_IN_TABLE_INSERT`, not `DIVIDE_BY_ZERO`.
   - When the child is no longer a `Cast` after optimization, Spark leaves the 
error alone (SPARK-42286).
   
   It is listed, unchecked, in #2314. The expression is the same in Spark 3.5 
through 4.2; only the message text changed in 4.0.
   
   ### Willingness to contribute
   
   I can contribute a fix for this bug independently
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to