FreakingMind opened a new pull request, #7661:
URL: https://github.com/apache/datafusion-comet/pull/7661

   ## Which issue does this PR close?
   
   Closes #7656.
   
   ## Rationale for this change
   
   When an INSERT narrows a numeric column, Spark wraps the cast in 
`CheckOverflowInTableInsert`. Comet had no serde for it, so the Project that 
feeds the write fell back to Spark, and with it the native Parquet write.
   
   The wrapper only changes the error. Spark rethrows any 
`SparkArithmeticException` from the cast as `CAST_OVERFLOW_IN_TABLE_INSERT`, 
which names the column instead of the value and has no query context. Passing 
the wrapper through would give `CAST_OVERFLOW` instead.
   
   ## What changes are included in this PR?
   
   - A native `CheckOverflowInTableInsert` that evaluates the cast and replaces 
an arithmetic error with the new `SparkError::CastOverflowInTableInsert`. It 
covers native errors, Comet's `CastOverflowException`, and a Java 
`SparkArithmeticException` from a child that runs through the JVM codegen 
dispatcher. Other errors pass through unchanged.
   - A serde that wraps only a `Cast` child. When the optimizer has rewritten 
the cast (SPARK-42286), Spark keeps the child's error, so the child is 
serialized on its own.
   - A converter case in both shims that calls 
`QueryExecutionErrors.castingCauseOverflowErrorInTableInsert`, so each Spark 
version gets its own message text (3.5 says "insert ... into", 4.x says "assign 
... to"). The proto carries the types as Spark SQL type names for this.
   - A line in `expressions.md`.
   
   There is one difference from Spark, which I documented. When an expression 
under the cast occurs more than once in the projection, for example `INSERT 
INTO t SELECT c, c FROM src`, Spark's subexpression elimination evaluates it 
outside the wrapper and reports the original error, such as `CAST_OVERFLOW`. 
Comet reports `CAST_OVERFLOW_IN_TABLE_INSERT` there, which is what Spark does 
with `spark.sql.subexpressionElimination.enabled=false`. Matching it would need 
the whole project list in the serde. The query fails either way; only the error 
class differs.
   
   Narrowing INSERTs now stay in Comet by default. The new expression gets the 
usual `spark.comet.expression.CheckOverflowInTableInsert.enabled` switch.
   
   ## How are these changes tested?
   
   New tests in `CometParquetWriterSuite`, each run with the Spark writer and 
with the native writer:
   
   - narrowing INSERTs into int, tinyint, smallint, bigint, decimal and a 
struct field, compared with the same INSERT run by Spark;
   - overflows, including boundaries, NaN, Infinity and decimal precision, plus 
`c div 0` under the cast, with the exception class, error class, SQLSTATE and 
message compared with Spark;
   - a non-arithmetic error under the cast, which keeps its class and query 
context;
   - the SPARK-42286 cases;
   - an overflowing literal with `ConstantFolding` excluded, which reaches the 
wrapper through the codegen dispatcher.
   
   They are Scala tests rather than SQL file tests because an INSERT is a 
`statement` there, which is not compared with Spark.
   
   All of them fail on main. I ran them on Spark 3.5, 4.0 and 4.1, together 
with the full `CometParquetWriterSuite` on 4.1. There is also a converter test 
in `SparkErrorConverterSuite` and Rust unit tests for the wrapper.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to