tlfua opened a new issue, #7549: URL: https://github.com/apache/datafusion-comet/issues/7549
### Describe the bug When the native HLL expressions fail, Comet raises a plain Comet execution error instead of Spark's `SparkRuntimeException` with the HLL error condition. Both engines fail on the same inputs, but the error class, SQLSTATE and message differ. | Expression | Spark error condition (SQLSTATE) | | --- | --- | | `hll_sketch_estimate`, `hll_union`, `hll_union_agg` on an invalid buffer | `HLL_INVALID_INPUT_SKETCH_BUFFER` (22546) | | `hll_union`, `hll_union_agg` with different lgConfigK and `allowDifferentLgConfigK = false` | `HLL_UNION_DIFFERENT_LG_K` (22000) | `native/common/src/error.rs` `SparkError` has no HLL variant, so these errors cannot reach `SparkErrorConverter` / `ShimSparkErrorConverter`. ### Steps to reproduce ```sql SELECT hll_sketch_estimate(X'0102030405'); ``` Spark 4.x raises `[HLL_INVALID_INPUT_SKETCH_BUFFER] Invalid call to hll_sketch_estimate; only valid HLL sketch buffers are supported as inputs ...` with SQLSTATE 22546. Comet raises a Comet execution error with a different message. ### Expected behavior Comet raises the same error condition and SQLSTATE as Spark. A likely fix, following https://github.com/apache/datafusion-comet/issues/5072 : - add `SparkError` variants for the two conditions in `native/common/src/error.rs`; - map them in the spark-4.x `ShimSparkErrorConverter` to `QueryExecutionErrors.hllInvalidInputSketchBuffer(function)` and `QueryExecutionErrors.hllUnionDifferentLgK(left, right, function)`; - cover them with `expect_error_class` SQL tests or `checkSparkError`. ### Additional context Surfaced by the HLL audit for https://github.com/apache/datafusion-comet/issues/7396 . The comparison experiment confirmed both engines fail on the same inputs ( https://github.com/apache/datafusion-comet/issues/7396#issuecomment-6703600973 ). The serdes currently list this as an incompatible reason that is never returned; the audit PR moves it to a compatibility note that links here. ### Willingness to contribute I can contribute a fix for this bug independently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
