andygrove opened a new issue, #4838: URL: https://github.com/apache/datafusion-comet/issues/4838
Triage pass over the open `requires-triage` queue, per the project [Bug Triage Guide](https://github.com/apache/datafusion-comet/blob/main/docs/source/contributor-guide/bug_triage.md). - Date: 2026-07-06 - Total issues processed: 26 (25 triaged, 1 skipped, 0 failed) - Type counts: 7 bugs, 18 enhancements - Priority counts applied: `priority:critical` 5, `priority:high` 2 - Guide: [docs/source/contributor-guide/bug_triage.md](https://github.com/apache/datafusion-comet/blob/main/docs/source/contributor-guide/bug_triage.md) Labels have already been applied and `requires-triage` removed from each issue listed under "Bugs" and "Enhancements". A reviewer should spot-check the calls and close this issue when satisfied. To correct a label, edit the affected issue directly. ## Bugs ### priority:critical - array_filter: array_compact fast path misfires when IsNotNull operand is not the lambda variable ([#4830](https://github.com/apache/datafusion-comet/issues/4830)) - Area labels: `area:expressions` - Rationale: `CometArrayFilter.convert` silently rewrites `filter(arr, x -> captured_col IS NOT NULL)` as `array_compact(arr)`, producing wrong results whenever `arr` has null elements or `captured_col` is null — decision-tree step 1 (silent wrong results). - Distinct-aggregate rewrite can split an incompatible-buffer aggregate across Comet and Spark, causing crash or wrong results ([#4813](https://github.com/apache/datafusion-comet/issues/4813)) - Area labels: `area:aggregation` - Rationale: incompatible partial buffers cross the Comet/Spark boundary in the 4-stage distinct rewrite; hard crash for `percentile_approx` but the reporter notes silent wrong results are likely for `collect_set`, `collect_list`, exact `percentile`, `avg`, decimal `sum`, and variance — escalated to critical under decision-tree step 1. - bug: `flatten` drops nulls and returns wrong results when a sub-array is null ([#4788](https://github.com/apache/datafusion-comet/issues/4788)) - Area labels: `area:expressions` - Rationale: reporter demonstrates Comet returns a non-null misaligned array where Spark returns `null`; silent data corruption over a plain Parquet scan — decision-tree step 1. - bug: `null IN ()` returns false instead of null under legacy null-in-empty behavior ([#4786](https://github.com/apache/datafusion-comet/issues/4786)) - Area labels: `area:expressions` - Rationale: Comet ignores `spark.sql.legacy.nullInEmptyListBehavior=true` and silently returns `false` where Spark returns `null` — decision-tree step 1. - [EPIC] Consistent handling of invalid UTF-8 in native StringType (ingress policy) ([#4764](https://github.com/apache/datafusion-comet/issues/4764)) - Area labels: none (existing `correctness`, `area:shuffle`, `area:scan`, `area:ffi` preserved by the reporter) - Rationale: reporter documents a latent-UB / silent-wrong-result hazard (arrow-rs `StringArray::value` on FFI-imported non-UTF-8 bytes is unchecked) plus the scan gap that Spark accepts and Comet rejects — decision-tree step 1 for the UB hazard; kept as an EPIC bug. ### priority:high - bug: array/map kernels crash on non-null child field via CometLocalTableScanExec ([#4789](https://github.com/apache/datafusion-comet/issues/4789)) - Area labels: `area:expressions`, `area:scan` - Rationale: native assertion failure / `InvalidArgumentError` panic on `slice`, `map_entries`, and `array_insert` when the local scan carries non-nullable child fields; visible crash rather than wrong results — decision-tree step 2. - bug: scalar subquery in RepartitionByExpression crashes with "Subquery N not found" ([#4787](https://github.com/apache/datafusion-comet/issues/4787)) - Area labels: `area:expressions` - Rationale: `CometNativeException: Subquery 75 not found for plan 12` when a scalar subquery appears in `DISTRIBUTE BY`; reproduces over plain Parquet — decision-tree step 2 (native crash). ## Enhancements - Support WindowGroupLimitExec (window-based limit pushdown) ([#4837](https://github.com/apache/datafusion-comet/issues/4837)) - Area labels: `area:expressions` - Rationale: request for a new native operator so plans containing `WindowGroupLimitExec` no longer fall back to Spark. - Support GROUPS window frame units ([#4836](https://github.com/apache/datafusion-comet/issues/4836)) - Area labels: `area:expressions` - Rationale: adds a currently unsupported window frame type; the existing behavior is a documented, clean fallback to Spark, not a defect. - Support FIRST_VALUE/LAST_VALUE on a RANGE window frame with a literal offset ([#4835](https://github.com/apache/datafusion-comet/issues/4835)) - Area labels: `area:expressions` - Rationale: extends native RANGE-frame coverage; today Comet correctly falls back to Spark, so this is a coverage gap. - Support RANGE window frames with an explicit offset on DATE or DECIMAL ORDER BY ([#4834](https://github.com/apache/datafusion-comet/issues/4834)) - Area labels: `area:expressions` - Rationale: documented current limitation with clean Spark fallback; request to broaden native RANGE-frame ORDER BY type support. - Partial project fallback: keep a projection/filter native by evaluating an unsupported subexpression in the JVM ([#4825](https://github.com/apache/datafusion-comet/issues/4825)) - Area labels: `area:expressions` - Rationale: proposes a new splitter hook so unsupported subexpressions no longer collapse the whole native island — a new feature behind an opt-in config. - [Enhancement] Enable mixed partial/final execution for approx_count_distinct (HyperLogLogPlusPlus) ([#4820](https://github.com/apache/datafusion-comet/issues/4820)) - Area labels: `area:aggregation` - Rationale: request to broaden native coverage by flipping `supportsMixedPartialFinal` for `approx_count_distinct` now that the buffer format matches Spark; not a defect. - Support grouping() and grouping_id() indicator functions ([#4814](https://github.com/apache/datafusion-comet/issues/4814)) - Area labels: `area:expressions` - Rationale: queries already run natively; the ask is to add test coverage and mark the functions as supported in the docs — a coverage/documentation enhancement. - feat: IO-level dynamic filter pushdown into the native Delta scan (delta-kernel-rs predicate reuse) ([#4809](https://github.com/apache/datafusion-comet/issues/4809)) - Area labels: `area:scan` - Rationale: performance follow-up to #4807 that adds IO-level pruning to the native Delta scan; new functionality. - feat: IO-level dynamic filter pushdown into IcebergScanExec (iceberg-rust predicate conversion) ([#4808](https://github.com/apache/datafusion-comet/issues/4808)) - Area labels: `area:scan` - Rationale: performance follow-up to #4807 that adds IO-level pruning inside iceberg-rust; new functionality. - feat: Runtime dynamic filter pushdown for native hash joins (build side -> probe side) ([#4807](https://github.com/apache/datafusion-comet/issues/4807)) - Area labels: none - Rationale: request to wire up DataFusion's dynamic filter machinery through Comet's native hash joins; new optimization behind an opt-in config. - Add optional fully-native deployment mode (Ballista-backed) ([#4796](https://github.com/apache/datafusion-comet/issues/4796)) - Area labels: none - Rationale: proposes a new deployment mode built on Ballista; discussion-phase feature request. - Support native batch session_window grouping (MergingSessionsExec / UpdatingSessionsExec) ([#4785](https://github.com/apache/datafusion-comet/issues/4785)) - Area labels: `area:aggregation` - Rationale: requests new native operators for batch session-window merging; today Comet cleanly falls back to Spark for this. - Performance optimizations for native in-memory cache (follow-on to #4591) ([#4781](https://github.com/apache/datafusion-comet/issues/4781)) - Area labels: `area:scan` - Rationale: follow-on performance work tracked from PR #4591 review; no correctness defect. - feat: Refactor shuffle partition writing behind a writer interface ([#4780](https://github.com/apache/datafusion-comet/issues/4780)) - Area labels: `area:shuffle` - Rationale: refactor to introduce a `PartitionWriter` abstraction so alternative shuffle backends can be added later; behavior-preserving. - Move CometCollationSuite into the spark-4.1+ test shim to avoid per-version duplication ([#4770](https://github.com/apache/datafusion-comet/issues/4770)) - Area labels: `area:ci` - Rationale: test/build-system cleanup to add `${shims.minorPlusVerSrc}` to `add-test-source` and move the suite; no behavior change. - Report unsupported metadata column names in fallback reasons ([#4757](https://github.com/apache/datafusion-comet/issues/4757)) - Area labels: `area:scan` - Rationale: improves fallback message to include the metadata column name; user-experience enhancement. - Math expressions: revisit interval / input-type gating and add regression tests once interval support lands ([#4756](https://github.com/apache/datafusion-comet/issues/4756)) - Area labels: `area:expressions` - Rationale: reporter explicitly states "no live defect today"; deferred work to add regression tests and revisit gating once native interval support lands. - Evaluate moving from chrono to Jiff ([#4754](https://github.com/apache/datafusion-comet/issues/4754)) - Area labels: `area:expressions` - Rationale: investigation into swapping the chrono dependency for Jiff since chrono-tz is being deprecated; enhancement / exploration. ## Escalations to consider - Distinct-aggregate rewrite can split an incompatible-buffer aggregate across Comet and Spark, causing crash or wrong results ([#4813](https://github.com/apache/datafusion-comet/issues/4813)) - Reporter observed a hard crash for `percentile_approx` but explicitly warns that other affected aggregates (`collect_set`, `collect_list`, exact `percentile`, decimal `sum`, `avg`, variance) risk silent wrong results — matches the guide's "priority:high crash also produces wrong results" escalation trigger; escalated to `priority:critical`. ## Skipped — needs more info - Bug triage results: 2026-06-29 ([#4751](https://github.com/apache/datafusion-comet/issues/4751)) - Prior triage summary issue (auto-labeled `requires-triage`); meta, awaiting human review and closure, not a bug. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
