andygrove opened a new issue, #7640: URL: https://github.com/apache/datafusion-comet/issues/7640
Bug triage pass for 2026-11-16. - Issues processed: 17 (8 bugs, 9 enhancements) - Bugs by priority: critical 2, high 0, medium 5, low 1 - Skipped: 1 Criteria: [Bug Triage Guide](https://github.com/apache/datafusion-comet/blob/main/docs/source/contributor-guide/bug_triage.md). Labels have already been applied and `requires-triage` removed. Please spot-check the calls and close this issue when satisfied. Many issues arrived already carrying a type label; those were kept as they were. ## Bugs ### priority:critical - Native MergeRows can panic or produce wrong masks - arrow 59 `and_not` mishandles sliced arrays ([#7596](https://github.com/apache/datafusion-comet/issues/7596)) - Area labels: none - Rationale: Besides the panic, `and_not` on a sliced array can return wrong masks with no error, so inserts and deletes land on the wrong rows, a silent wrong result. - Native string to decimal cast fails on a string with more than 38 digits: NULL or `CAST_INVALID_INPUT` where Spark returns a value ([#7581](https://github.com/apache/datafusion-comet/issues/7581)) - Area labels: `area:expressions` - Rationale: Comet returns NULL where Spark returns a value (for example `1.00`) with ANSI off, a silent wrong result. ### priority:medium - fair_unified: consumers that have not started yet take an equal share, so the running sort or aggregate spills early ([#7628](https://github.com/apache/datafusion-comet/issues/7628)) - Area labels: none - Rationale: Results are correct, but the extra spilling is a significant performance degradation (56 GB spilled against 19.7 GB for Spark in the report). - make_date can throw on unconsumed rows through PySpark and raw row iterators ([#7616](https://github.com/apache/datafusion-comet/issues/7616)) - Area labels: `area:expressions` - Rationale: Comet raises an error on a row Spark never evaluates, only for partial consumers of raw row iterators; a visible failure, not a wrong result. - A failed native `grow` is only logged when the task makes no later `try_grow` ([#7615](https://github.com/apache/datafusion-comet/issues/7615)) - Area labels: none - Rationale: A memory request Spark rejected is not surfaced, so the task can run past its grant; it needs a failed `grow` and no later `try_grow`, and does not return wrong data. - Native shuffle write fails when the local dir path contains `.data`; failed tasks also leave temp files behind ([#7594](https://github.com/apache/datafusion-comet/issues/7594)) - Area labels: `area:shuffle` - Rationale: Native shuffle write fails for local dirs whose path contains `.data`, which can be worked around by renaming the directory, and failed tasks leave temp files behind. - lpad/rpad do not use withNullShortCircuit, so ANSI errors are raised on rows Spark skips ([#7583](https://github.com/apache/datafusion-comet/issues/7583)) - Area labels: `area:expressions` - Rationale: Under ANSI, Comet raises an error on a NULL-input row where Spark returns NULL; a visible failure that needs an ANSI cast in a child expression. ### priority:low - Native string to decimal cast raises `CAST_INVALID_INPUT` instead of `NUMERIC_VALUE_OUT_OF_RANGE` under ANSI when the scale adjustment is more than 38 ([#7576](https://github.com/apache/datafusion-comet/issues/7576)) - Area labels: `area:expressions` - Rationale: Both engines fail under ANSI and return NULL otherwise; only the error class differs. ## Enhancements - Native Iceberg writes: assert manifest metrics parity with iceberg-java for every data type ([#7639](https://github.com/apache/datafusion-comet/issues/7639)) - Area labels: `area:writer` - Rationale: Asks for more parity tests before native Iceberg writes are enabled by default. - Native Iceberg writes: support `hdfs` data locations ([#7637](https://github.com/apache/datafusion-comet/issues/7637)) - Area labels: `area:writer` - Rationale: Native Iceberg writes fall back to iceberg-java for `hdfs` by design; this asks for new support. - Support struct and array map keys in native MapSort ([#7632](https://github.com/apache/datafusion-comet/issues/7632)) - Area labels: `area:expressions` - Rationale: Requests native evaluation for key types that already fall back. - Comet Native Executors: raise Spark's exceptions for cluster failures ([#7621](https://github.com/apache/datafusion-comet/issues/7621)) - Area labels: none - Rationale: Requests a new structured error path; the limitation is already documented. - Migrate to the new apache/datafusion-spark repository ([#7619](https://github.com/apache/datafusion-comet/issues/7619)) - Area labels: none - Rationale: Docs, skills and dependency housekeeping after an upstream move; `main` is unaffected. - Research using the Rust Allocator API (stabilized in 1.100) for memory-pool accounting ([#7613](https://github.com/apache/datafusion-comet/issues/7613)) - Area labels: none - Rationale: A research task, not a defect. - Restore native row-group pruning for FLOAT and DOUBLE range filters in Iceberg scans ([#7588](https://github.com/apache/datafusion-comet/issues/7588)) - Area labels: `area:scan` - Rationale: Results are correct today; this asks to recover pruning that #7156 turned off. - Speculate string functions such as upper in NullShortCircuit by adding them to is_cheap ([#7587](https://github.com/apache/datafusion-comet/issues/7587)) - Area labels: `area:expressions` - Rationale: A performance optimization request. - Consolidate the Scala "cannot fail" checks: arrays.cannotThrow and QueryPlanSerde.cannotFail ([#7585](https://github.com/apache/datafusion-comet/issues/7585)) - Area labels: `area:expressions` - Rationale: A refactor to remove duplicated lists. ## Escalations to consider - lpad/rpad do not use withNullShortCircuit, so ANSI errors are raised on rows Spark skips ([#7583](https://github.com/apache/datafusion-comet/issues/7583)) - Same family as #7616 and the #7456 work: Comet raises errors on rows Spark skips. If more reports arrive under ANSI, consider `priority:high`. ## Skipped — needs more info - TPC-DS q54/q68: local SF100 results and distributed reproduction limits ([#6946](https://github.com/apache/datafusion-comet/issues/6946)) - A reproduction report for #6418 that says it is not a claim of a defect; unclear whether it is a bug or an enhancement. `requires-triage` is left in place. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
