FreakingMind opened a new pull request, #7661: URL: https://github.com/apache/datafusion-comet/pull/7661
## Which issue does this PR close? Closes #7656. ## Rationale for this change When an INSERT narrows a numeric column, Spark wraps the cast in `CheckOverflowInTableInsert`. Comet had no serde for it, so the Project that feeds the write fell back to Spark, and with it the native Parquet write. The wrapper only changes the error. Spark rethrows any `SparkArithmeticException` from the cast as `CAST_OVERFLOW_IN_TABLE_INSERT`, which names the column instead of the value and has no query context. Passing the wrapper through would give `CAST_OVERFLOW` instead. ## What changes are included in this PR? - A native `CheckOverflowInTableInsert` that evaluates the cast and replaces an arithmetic error with the new `SparkError::CastOverflowInTableInsert`. It covers native errors, Comet's `CastOverflowException`, and a Java `SparkArithmeticException` from a child that runs through the JVM codegen dispatcher. Other errors pass through unchanged. - A serde that wraps only a `Cast` child. When the optimizer has rewritten the cast (SPARK-42286), Spark keeps the child's error, so the child is serialized on its own. - A converter case in both shims that calls `QueryExecutionErrors.castingCauseOverflowErrorInTableInsert`, so each Spark version gets its own message text (3.5 says "insert ... into", 4.x says "assign ... to"). The proto carries the types as Spark SQL type names for this. - A line in `expressions.md`. There is one difference from Spark, which I documented. When an expression under the cast occurs more than once in the projection, for example `INSERT INTO t SELECT c, c FROM src`, Spark's subexpression elimination evaluates it outside the wrapper and reports the original error, such as `CAST_OVERFLOW`. Comet reports `CAST_OVERFLOW_IN_TABLE_INSERT` there, which is what Spark does with `spark.sql.subexpressionElimination.enabled=false`. Matching it would need the whole project list in the serde. The query fails either way; only the error class differs. Narrowing INSERTs now stay in Comet by default. The new expression gets the usual `spark.comet.expression.CheckOverflowInTableInsert.enabled` switch. ## How are these changes tested? New tests in `CometParquetWriterSuite`, each run with the Spark writer and with the native writer: - narrowing INSERTs into int, tinyint, smallint, bigint, decimal and a struct field, compared with the same INSERT run by Spark; - overflows, including boundaries, NaN, Infinity and decimal precision, plus `c div 0` under the cast, with the exception class, error class, SQLSTATE and message compared with Spark; - a non-arithmetic error under the cast, which keeps its class and query context; - the SPARK-42286 cases; - an overflowing literal with `ConstantFolding` excluded, which reaches the wrapper through the codegen dispatcher. They are Scala tests rather than SQL file tests because an INSERT is a `statement` there, which is not compared with Spark. All of them fail on main. I ran them on Spark 3.5, 4.0 and 4.1, together with the full `CometParquetWriterSuite` on 4.1. There is also a converter test in `SparkErrorConverterSuite` and Rust unit tests for the wrapper. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
