andygrove opened a new issue, #7640:
URL: https://github.com/apache/datafusion-comet/issues/7640

   Bug triage pass for 2026-11-16.
   
   - Issues processed: 17 (8 bugs, 9 enhancements)
   - Bugs by priority: critical 2, high 0, medium 5, low 1
   - Skipped: 1
   
   Criteria: [Bug Triage 
Guide](https://github.com/apache/datafusion-comet/blob/main/docs/source/contributor-guide/bug_triage.md).
   
   Labels have already been applied and `requires-triage` removed. Please 
spot-check the calls and close this issue when satisfied. Many issues arrived 
already carrying a type label; those were kept as they were.
   
   ## Bugs
   
   ### priority:critical
   
   - Native MergeRows can panic or produce wrong masks - arrow 59 `and_not` 
mishandles sliced arrays 
([#7596](https://github.com/apache/datafusion-comet/issues/7596))
     - Area labels: none
     - Rationale: Besides the panic, `and_not` on a sliced array can return 
wrong masks with no error, so inserts and deletes land on the wrong rows, a 
silent wrong result.
   - Native string to decimal cast fails on a string with more than 38 digits: 
NULL or `CAST_INVALID_INPUT` where Spark returns a value 
([#7581](https://github.com/apache/datafusion-comet/issues/7581))
     - Area labels: `area:expressions`
     - Rationale: Comet returns NULL where Spark returns a value (for example 
`1.00`) with ANSI off, a silent wrong result.
   
   ### priority:medium
   
   - fair_unified: consumers that have not started yet take an equal share, so 
the running sort or aggregate spills early 
([#7628](https://github.com/apache/datafusion-comet/issues/7628))
     - Area labels: none
     - Rationale: Results are correct, but the extra spilling is a significant 
performance degradation (56 GB spilled against 19.7 GB for Spark in the report).
   - make_date can throw on unconsumed rows through PySpark and raw row 
iterators ([#7616](https://github.com/apache/datafusion-comet/issues/7616))
     - Area labels: `area:expressions`
     - Rationale: Comet raises an error on a row Spark never evaluates, only 
for partial consumers of raw row iterators; a visible failure, not a wrong 
result.
   - A failed native `grow` is only logged when the task makes no later 
`try_grow` ([#7615](https://github.com/apache/datafusion-comet/issues/7615))
     - Area labels: none
     - Rationale: A memory request Spark rejected is not surfaced, so the task 
can run past its grant; it needs a failed `grow` and no later `try_grow`, and 
does not return wrong data.
   - Native shuffle write fails when the local dir path contains `.data`; 
failed tasks also leave temp files behind 
([#7594](https://github.com/apache/datafusion-comet/issues/7594))
     - Area labels: `area:shuffle`
     - Rationale: Native shuffle write fails for local dirs whose path contains 
`.data`, which can be worked around by renaming the directory, and failed tasks 
leave temp files behind.
   - lpad/rpad do not use withNullShortCircuit, so ANSI errors are raised on 
rows Spark skips 
([#7583](https://github.com/apache/datafusion-comet/issues/7583))
     - Area labels: `area:expressions`
     - Rationale: Under ANSI, Comet raises an error on a NULL-input row where 
Spark returns NULL; a visible failure that needs an ANSI cast in a child 
expression.
   
   ### priority:low
   
   - Native string to decimal cast raises `CAST_INVALID_INPUT` instead of 
`NUMERIC_VALUE_OUT_OF_RANGE` under ANSI when the scale adjustment is more than 
38 ([#7576](https://github.com/apache/datafusion-comet/issues/7576))
     - Area labels: `area:expressions`
     - Rationale: Both engines fail under ANSI and return NULL otherwise; only 
the error class differs.
   
   ## Enhancements
   
   - Native Iceberg writes: assert manifest metrics parity with iceberg-java 
for every data type 
([#7639](https://github.com/apache/datafusion-comet/issues/7639))
     - Area labels: `area:writer`
     - Rationale: Asks for more parity tests before native Iceberg writes are 
enabled by default.
   - Native Iceberg writes: support `hdfs` data locations 
([#7637](https://github.com/apache/datafusion-comet/issues/7637))
     - Area labels: `area:writer`
     - Rationale: Native Iceberg writes fall back to iceberg-java for `hdfs` by 
design; this asks for new support.
   - Support struct and array map keys in native MapSort 
([#7632](https://github.com/apache/datafusion-comet/issues/7632))
     - Area labels: `area:expressions`
     - Rationale: Requests native evaluation for key types that already fall 
back.
   - Comet Native Executors: raise Spark's exceptions for cluster failures 
([#7621](https://github.com/apache/datafusion-comet/issues/7621))
     - Area labels: none
     - Rationale: Requests a new structured error path; the limitation is 
already documented.
   - Migrate to the new apache/datafusion-spark repository 
([#7619](https://github.com/apache/datafusion-comet/issues/7619))
     - Area labels: none
     - Rationale: Docs, skills and dependency housekeeping after an upstream 
move; `main` is unaffected.
   - Research using the Rust Allocator API (stabilized in 1.100) for 
memory-pool accounting 
([#7613](https://github.com/apache/datafusion-comet/issues/7613))
     - Area labels: none
     - Rationale: A research task, not a defect.
   - Restore native row-group pruning for FLOAT and DOUBLE range filters in 
Iceberg scans ([#7588](https://github.com/apache/datafusion-comet/issues/7588))
     - Area labels: `area:scan`
     - Rationale: Results are correct today; this asks to recover pruning that 
#7156 turned off.
   - Speculate string functions such as upper in NullShortCircuit by adding 
them to is_cheap 
([#7587](https://github.com/apache/datafusion-comet/issues/7587))
     - Area labels: `area:expressions`
     - Rationale: A performance optimization request.
   - Consolidate the Scala "cannot fail" checks: arrays.cannotThrow and 
QueryPlanSerde.cannotFail 
([#7585](https://github.com/apache/datafusion-comet/issues/7585))
     - Area labels: `area:expressions`
     - Rationale: A refactor to remove duplicated lists.
   
   ## Escalations to consider
   
   - lpad/rpad do not use withNullShortCircuit, so ANSI errors are raised on 
rows Spark skips 
([#7583](https://github.com/apache/datafusion-comet/issues/7583))
     - Same family as #7616 and the #7456 work: Comet raises errors on rows 
Spark skips. If more reports arrive under ANSI, consider `priority:high`.
   
   ## Skipped — needs more info
   
   - TPC-DS q54/q68: local SF100 results and distributed reproduction limits 
([#6946](https://github.com/apache/datafusion-comet/issues/6946))
     - A reproduction report for #6418 that says it is not a claim of a defect; 
unclear whether it is a bug or an enhancement. `requires-triage` is left in 
place.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to