andygrove opened a new issue, #6077: URL: https://github.com/apache/datafusion-comet/issues/6077
Triage pass over the open `requires-triage` queue, per the project [Bug Triage Guide](https://github.com/apache/datafusion-comet/blob/main/docs/source/contributor-guide/bug_triage.md). - Date: 2026-09-21 - Total issues processed: 57 (50 triaged, 7 skipped, 0 failed) - Type counts: 15 bugs, 35 enhancements - Priority counts applied: `priority:critical` 5, `priority:high` 1, `priority:medium` 6, `priority:low` 3 - Guide: [docs/source/contributor-guide/bug_triage.md](https://github.com/apache/datafusion-comet/blob/main/docs/source/contributor-guide/bug_triage.md) Labels have already been applied. A reviewer should spot-check the calls below and close this issue when satisfied; corrections should be made directly on the affected issue. Notes on this pass: - **Three type reclassifications are worth a second look**, all from `enhancement` to `bug`: [#6024](https://github.com/apache/datafusion-comet/issues/6024), [#6022](https://github.com/apache/datafusion-comet/issues/6022) and [#6060](https://github.com/apache/datafusion-comet/issues/6060). The first two were filed by their author with the feature-request template and an `enhancement` label already set. They were reclassified because in both the query *fails* under Comet where plain Spark succeeds, which the guide places under `priority:medium` "broken features that have workarounds" rather than under new functionality. If the intent was to track them as new resilience features, flipping them back is a one-line edit. - **One priority escalation**: [#6002](https://github.com/apache/datafusion-comet/issues/6002) was raised from the author's `priority:high` to `priority:critical`; see *Escalations to consider*. - **The guide's area table is out of date.** The repository now carries `area:memory`, `area:joins` and `area:Iceberg`, none of which appear in the guide, so this pass did not apply them — the same note was made in the 2026-08-17, 2026-08-31 and 2026-09-14 passes. That gap cost real labelling this time: [#5997](https://github.com/apache/datafusion-comet/issues/5997), [#5961](https://github.com/apache/datafusion-comet/issues/5961), [#6028](https://github.com/apache/datafusion-comet/issues/6028) and [#6032](https://github.com/apache/datafusion-comet/issues/6032) have no area label but are plainly `area:memory`, and [#6056](https://github.com/apache/datafusion-comet/issues/6056), [#6013](https://github.com/apache/datafusion-comet/issues/6013), [#6008](https://github.com/apache/datafusion-comet/issues/6008) and [#5962](https://github.com/apache/datafusion-comet/issues/5962) are plainly `area:joins`. Adding the three labels to the guide's area table would fix this for the next pass. - **The `regression` label now exists in the repository but is not in the guide**, so this pass did not apply it. That is the label proposed by [#5925](https://github.com/apache/datafusion-comet/issues/5925); the remaining work in that issue is the guide section and the skill step, which is why it is still open and classified as an enhancement here. - The guide lists `spark 4` as a pre-existing area indicator, but the repository only has `spark 4.0` / `spark 4.1` / `spark 4.2` / `spark 3.x`, so no `spark 4` label was applied. - Where an issue already carried non-guide labels (`correctness`, `performance`, `documentation`, `area:memory`, `area:shuffle`), those were left untouched and are included in the area lists below for context. - No enhancement in this queue arrived with an author-applied priority label, so nothing needed stripping. ## Bugs ### priority:critical - Variance and standard deviation return incorrect results for large nearby values ([#6044](https://github.com/apache/datafusion-comet/issues/6044)) - Area labels: `area:aggregation` - Rationale: `var_pop` returns 2.0 where Spark returns 1.0, a 100% relative error with no error or fallback, which is step 1 of the decision tree. - Sliding-window SUM(BIGINT) ignores ANSI and TRY overflow semantics ([#6043](https://github.com/apache/datafusion-comet/issues/6043)) - Area labels: `area:aggregation` - Rationale: `try_sum` returns a wrapped negative value where Spark returns NULL and ANSI `sum` returns that value instead of raising, so the wrong answer is delivered silently in both modes. - Nested floating-point IN membership does not match Spark for signed zero ([#6019](https://github.com/apache/datafusion-comet/issues/6019)) - Area labels: `area:expressions` - Rationale: the native nested membership comparator returns `false` where Spark's interpreted ordering returns `true` for `-0.0` versus `0.0`, with no error, which is the guide's silent-wrong-results case. - SUM(decimal) returns NULL or throws under ANSI when an intermediate sum overflows the buffer precision but the final result fits ([#6002](https://github.com/apache/datafusion-comet/issues/6002)) - Area labels: `area:aggregation`, `correctness` - Rationale: in legacy mode Comet returns NULL where Spark returns `0.6`, a silent wrong result at step 1 of the decision tree; escalated from the author's `priority:high` (see escalations below). - Iceberg serde: residual reflection failures can silently drop residual filtering ([#5992](https://github.com/apache/datafusion-comet/issues/5992)) - Area labels: `area:scan` - Rationale: an unexpected reflection failure is indistinguishable from an intentional non-push, so the task is serialized without the residual predicate and can return extra rows with no error — step 1 asks whether the bug *can* cause silent wrong results, and here it can whenever no exact filter sits above the scan. ### priority:high - TRY_CAST on narrowing map keys fails where Spark returns a map with a null key ([#5995](https://github.com/apache/datafusion-comet/issues/5995)) - Area labels: `area:expressions` - Rationale: on `main` the non-nullable key field makes `StructArray::new` panic inside arrow and abort the executor, which is the guide's step 2 crash case on a plan Comet itself admitted as `Compatible`. ### priority:medium - Native S3 scan on EKS/IRSA turns a transient STS throttle into a hard 403 storm ([#6024](https://github.com/apache/datafusion-comet/issues/6024)) - Area labels: `area:scan` - Rationale: the credential chain does not retry the throttled assume-role call and silently downgrades to the node instance role, so the job dies with 403s where plain Spark's S3A retry survives; loud rather than silent, and workable around by lowering per-pod concurrency, so step 3 rather than step 1 or 2. Reclassified from `enhancement`. - Native Parquet S3 scan fails on credential provider classes that Spark/Hadoop accept ([#6022](https://github.com/apache/datafusion-comet/issues/6022)) - Area labels: `area:scan` - Rationale: the query fails with `Unsupported credential provider` where plain Spark runs it fine, and Comet's own Iceberg path succeeds on the same cluster — a broken feature with a workaround, which the guide's `priority:medium` row covers under "missing expression support, writer feature gaps". Reclassified from `enhancement`. - Legacy mode negation of a scalar signed minimum errors where the array path wraps ([#6015](https://github.com/apache/datafusion-comet/issues/6015)) - Area labels: `area:expressions` - Rationale: the scalar arm raises an arithmetic overflow error in legacy mode where Spark and Comet's own array arm both wrap, failing the query rather than returning a wrong value, so step 3. - Columnar shuffle rejects range partitioning based on a native-serde check it never uses ([#5971](https://github.com/apache/datafusion-comet/issues/5971)) - Area labels: `area:shuffle` - Rationale: the reporter states it is an unnecessary-fallback bug rather than a correctness bug — the serialized `SortOrder` is never sent to native code on the columnar path, so Comet's shuffle is disabled for orderings the JVM would have partitioned correctly. - fair_unified memory pool caps the pool total at pool_size / num_consumers instead of each consumer's share ([#5961](https://github.com/apache/datafusion-comet/issues/5961)) - Area labels: `area:memory` - Rationale: the default off-heap pool grants a whole task only `pool_size / num_consumers` and tightens as consumers register, so work spills or fails early; a functional regression in the default configuration with a workaround (select another pool). Introduced by the DataFusion 53 upgrade (#3629), which predates the `1.0.0` tag, so it is not a regression against the most recent release. - Field id gating differs from Spark: root-only check and no dependence on fieldId.read.enabled ([#5936](https://github.com/apache/datafusion-comet/issues/5936)) - Area labels: `area:scan` - Rationale: two divergences, neither of which is an established wrong value — Comet returns name-matched rows where Spark raises, and Comet rejects a file whose ids are only on nested fields where Spark null-fills. Kept at step 3; escalate if someone shows the rows returned in the first case differ from the stored data. ### priority:low - Parquet writer tests can capture a previous write's plan ([#6075](https://github.com/apache/datafusion-comet/issues/6075)) - Area labels: `area:writer` - Rationale: `captureWritePlan` accepts a delayed callback from a setup write, producing a false assertion failure; test-only, which is step 4. - ci: a transient Maven Central failure in lint-java is reported as a scalafix violation ([#6060](https://github.com/apache/datafusion-comet/issues/6060)) - Area labels: `area:ci` - Rationale: a plugin-resolution hiccup fails the job and turns `Required Checks` red with no source file at fault; CI tooling, so step 4, but see escalations. - Nightly Miri workflow has failed every night since 2026-07-01 ([#5999](https://github.com/apache/datafusion-comet/issues/5999)) - Area labels: `area:ci` - Rationale: 79 consecutive scheduled-run failures, none of them in Comet's own `unsafe` code and none blocking a pull request; the cost is a lost signal rather than a broken build, which is step 4. ## Enhancements - Add native Vortex scan support ([#6070](https://github.com/apache/datafusion-comet/issues/6070)) - Area labels: `area:scan` - Rationale: proposes a new optional contrib module and native reader for a format Comet does not read today. - Remove memory accounting from Comet's on-heap mode ([#6063](https://github.com/apache/datafusion-comet/issues/6063)) - Area labels: `area:shuffle`, `area:memory` - Rationale: removal of a second accounting implementation that protects nothing — a refactor, with no reported wrong behaviour in the supported off-heap configuration. - Native Iceberg scan: support abfs/abfss (Azure Data Lake Storage Gen2) locations ([#6058](https://github.com/apache/datafusion-comet/issues/6058)) - Area labels: `area:scan` - Rationale: the scheme is declined at planning time and the scan runs on the JVM exactly as without the plugin, so this is new coverage rather than a defect; the backend is already compiled in. - Allow row-based build sides (such as JDBC) to convert to Arrow for native BroadcastHashJoin ([#6056](https://github.com/apache/datafusion-comet/issues/6056)) - Area labels: none (`area:joins` would apply if the guide adopted it) - Rationale: asks for a new row-to-Arrow bridge at the broadcast boundary; today's behaviour is a correct fallback, not a defect. - Audit Comet logical-link repair across AQE replanning ([#6034](https://github.com/apache/datafusion-comet/issues/6034)) - Area labels: none - Rationale: a design audit that explicitly "does not assume all repair is removable or establish another failing exchange case". - Tuning guide does not explain that Comet's memory comes out of the executor container ([#6032](https://github.com/apache/datafusion-comet/issues/6032)) - Area labels: `documentation` (`area:memory` would apply if the guide adopted it) - Rationale: documentation addition, which the guide's type table places under `enhancement`. - Memory management cgroup diagram does not show which config value sizes each region ([#6028](https://github.com/apache/datafusion-comet/issues/6028)) - Area labels: none (`area:memory` would apply if the guide adopted it) - Rationale: documentation improvement to an existing diagram. - perf: broadcast hash join rebuilds the build-side hash table in every task ([#6013](https://github.com/apache/datafusion-comet/issues/6013)) - Area labels: `performance` (`area:joins` would apply if the guide adopted it) - Rationale: a performance optimization request — the results are correct, the work just scales with task count instead of executor count. - Evaluate rotating the native shuffle spill file at a byte threshold ([#6010](https://github.com/apache/datafusion-comet/issues/6010)) - Area labels: `area:shuffle` - Rationale: asks for a measured comparison of three spill-file layouts; no failure is reported against the current one. - Free and account for spill range metadata in the native shuffle writer ([#6009](https://github.com/apache/datafusion-comet/issues/6009)) - Area labels: `area:shuffle` - Rationale: a memory-accounting and hygiene improvement with a computed worst case and no observed failure, consistent with how the other accounting-gap issues in this queue (#5997, #6063) are classified. - Text-format source on a broadcast build side blocks native `BroadcastHashJoin` ([#6008](https://github.com/apache/datafusion-comet/issues/6008)) - Area labels: none (`area:joins` would apply if the guide adopted it) - Rationale: asks either for auto-inserted `CometSparkToColumnar` on broadcast build sides or a native Text reader; both are new capability, and the issue notes the interim config workaround already works. - Track evaluation masks for data-dependent errors beyond unbase64 operator fallback ([#6006](https://github.com/apache/datafusion-comet/issues/6006)) - Area labels: `area:expressions` - Rationale: explicitly "does not claim that every expression/context combination below is currently broken" — it tracks the audit that would establish which, if any, are. - Global aggregate with no grouping keys and no aggregate functions falls back to Spark ([#6001](https://github.com/apache/datafusion-comet/issues/6001)) - Area labels: `area:aggregation` - Rationale: a correct fallback with a `[COMET: ...]` reason, costing coverage and two transitions rather than correctness. - Improve memory accounting: bound the JVM Arrow allocator and make native reclaim work ([#5997](https://github.com/apache/datafusion-comet/issues/5997)) - Area labels: none (`area:memory` would apply if the guide adopted it) - Rationale: collects four accounting improvements ordered by cost, after #5993 established that thresholding a cgroup counter does not work. - Support Spark-compatible native hash and xxhash64 for decimals with precision >18 ([#5994](https://github.com/apache/datafusion-comet/issues/5994)) - Area labels: `area:expressions`, `performance` - Rationale: the SQL admission guard already rejects these types recursively, so results are protected today and this is new native coverage. - Nightly SNAPSHOT publishing is blocked until ASF Infra provisions Nexus credentials ([#5985](https://github.com/apache/datafusion-comet/issues/5985)) - Area labels: `area:ci` - Rationale: tracks the completion of the publishing feature added in #5902 once INFRA-28409 is resolved; the 401 is the absence of credentials, not a defect in the workflow. - [Variant] Preserve typed-value precedence without residual pre-rewriting ([#5980](https://github.com/apache/datafusion-comet/issues/5980)) - Area labels: `area:scan` - Rationale: moves an existing, working decision earlier in the pipeline so the temporary null bitmap and replacement residual column from #5868 can be removed. - [Variant] Replace missing-key metadata repair with native permissive reconstruction ([#5979](https://github.com/apache/datafusion-comet/issues/5979)) - Area labels: `area:scan` - Rationale: replaces the `extend_shredded_metadata` workaround with a writable output dictionary; the Arrow panic it cites is guarded by the current repair. - [Variant] Remove the extra Spark byte-reconstruction pass ([#5978](https://github.com/apache/datafusion-comet/issues/5978)) - Area labels: `area:scan` - Rationale: folds a second pass into the original traversal — a performance and structure change over behaviour that is already correct. - [Variant] Consolidate Spark-compatible missing-value validation and errors ([#5977](https://github.com/apache/datafusion-comet/issues/5977)) - Area labels: `area:scan` - Rationale: moves existing checks into the recursive unshredder behind an opt-in policy, keeping Arrow's specification-conforming default. - Audit remaining native Partial-to-Spark Final aggregate compatibility ([#5975](https://github.com/apache/datafusion-comet/issues/5975)) - Area labels: `area:aggregation` - Rationale: recovers native coverage that #5421 deliberately withdrew; the withdrawal is the safe state, so this is coverage work rather than a defect. - ci: cancel merge queue runs on the first job failure ([#5969](https://github.com/apache/datafusion-comet/issues/5969)) - Area labels: `area:ci` - Rationale: a CI efficiency improvement — the queue already reaches the right verdict, it just wastes roughly an hour of runner time doing so. - test: cover Java float rendering through SQL file tests and consolidate the Iceberg partition path suites ([#5968](https://github.com/apache/datafusion-comet/issues/5968)) - Area labels: `area:expressions` - Rationale: two deferred test-placement suggestions from the #5840 review that restructure tests rather than change behaviour. - Forward Spark-generated runtime Bloom predicates into native Parquet readers ([#5965](https://github.com/apache/datafusion-comet/issues/5965)) - Area labels: `area:scan` - Rationale: the issue states it is an optimization gap and that the existing residual filter and join preserve query results. - Investigate mixed-type duplicate Parquet roots: Spark field-ID reads return anomalous values ([#5964](https://github.com/apache/datafusion-comet/issues/5964)) - Area labels: `area:scan` - Rationale: the anomalous `[1,0,null]` is Spark's own output, and the contrasting Comet result came from an unpublished experimental candidate, so no defect in released Comet is established; this tracks the investigation that would define the target behaviour. - Compute safe reusable expressions before broadcast materialization ([#5962](https://github.com/apache/datafusion-comet/issues/5962)) - Area labels: `area:expressions` (`area:joins` would also apply if the guide adopted it) - Rationale: a planning optimization with an explicit no-change path for unsupported cases. - Delegate Delta physical schema preparation to DeltaParquetFileFormat.prepareSchemaForRead ([#5949](https://github.com/apache/datafusion-comet/issues/5949)) - Area labels: `area:scan` - Rationale: removes duplication of logic Delta already implements; a follow-up from the #5365 review with the existing fallbacks staying in place. - Route Delta deletion-vector metadata preparation through the scan's instrumented reader ([#5948](https://github.com/apache/datafusion-comet/issues/5948)) - Area labels: `area:scan` - Rationale: the shared metadata cache is filled correctly; what is missing is that the preparation I/O lands in the scan's counters, which is an instrumentation improvement. - Add envelope and provider contract tests for the Delta plan-data injector ([#5947](https://github.com/apache/datafusion-comet/issues/5947)) - Area labels: `area:scan` - Rationale: adds tests to pin an existing boundary; marked non-blocking in the #5365 review. - Make the Delta contrib's file-planning boundary explicit ([#5946](https://github.com/apache/datafusion-comet/issues/5946)) - Area labels: `area:scan` - Rationale: turns an implicit contract into an explicit one with regression tests; no wrong partitions are reported. - Reuse the common native scan builder for ordinary columns in the Delta deletion-vector path ([#5945](https://github.com/apache/datafusion-comet/issues/5945)) - Area labels: `area:scan` - Rationale: de-duplication so a later serializer fix reaches both paths — a maintainability improvement. - Share the native-reader admission checks between the core Parquet scan and the Delta contrib ([#5944](https://github.com/apache/datafusion-comet/issues/5944)) - Area labels: `area:scan` - Rationale: the three checks agree today; the concern is that a future fix to any of them would need a separate Delta update. - Native Delta scan: derive the task injection key from a stable scan identity ([#5943](https://github.com/apache/datafusion-comet/issues/5943)) - Area labels: `area:scan` - Rationale: the reporter states that exchange boundaries isolate the common self-join shapes today, making this a key-contract hardening rather than an observed wrong answer; worth revisiting as a bug if a reachable collision is found. - Remove the legacy ShuffleWriter.output_data_file field from the native shuffle plan ([#5927](https://github.com/apache/datafusion-comet/issues/5927)) - Area labels: `area:shuffle` - Rationale: removes a proto field kept for native binaries the versioning policy no longer allows; dead-code cleanup. - Track whether a bug is a regression from the most recent release ([#5925](https://github.com/apache/datafusion-comet/issues/5925)) - Area labels: none - Rationale: a triage-process improvement. The `regression` label it proposes now exists in the repository; the guide section and the `bug-triage` skill step do not, which is the remaining work. ## Escalations to consider - SUM(decimal) returns NULL or throws under ANSI when an intermediate sum overflows the buffer precision but the final result fits ([#6002](https://github.com/apache/datafusion-comet/issues/6002)) - Escalated `priority:high` to `priority:critical` in this pass. In legacy mode Comet returns NULL where Spark returns `0.6`, with the aggregate running entirely in Comet and no mixed plan required. The guide's first prioritization principle is "correctness over crashes", and step 1 of the decision tree puts a silent wrong result at critical regardless of whether the ANSI variant raises. - Native S3 scan on EKS/IRSA turns a transient STS throttle into a hard 403 storm ([#6024](https://github.com/apache/datafusion-comet/issues/6024)) - Left at `priority:medium`, but it matches the guide's trigger "a `priority:medium` bug ... affects a common workload → consider escalating to `priority:high`". EKS with IRSA against S3-backed tables is a mainstream deployment, and the reported incident lost 238 of 250 executors at once on a real run. - ci: a transient Maven Central failure in lint-java is reported as a scalafix violation ([#6060](https://github.com/apache/datafusion-comet/issues/6060)) - Left at `priority:low`. The guide's trigger is "a `priority:low` CI flake is blocking PR merges consistently → escalate to `priority:medium`". One occurrence is documented, so the trigger is not met; if the `Lint Java` matrix is seen failing this way again, it should move to medium, since the fix is already modelled by the adjacent `scalafix-syntactic` job. ## Skipped — needs more info All seven are tracking or record issues rather than bug reports or feature requests, so no `bug`/`enhancement` label fits and `requires-triage` was left in place. Six of them are the output of this same process. They will reappear in every future pass until they are closed or the queue query excludes them; that is worth fixing, either by closing them once reviewed or by having the skill filter titles beginning "Bug triage results:". - Bug triage results: 2026-09-14 ([#5922](https://github.com/apache/datafusion-comet/issues/5922)) - The summary issue from the previous pass, awaiting reviewer sign-off. `requires-triage` was applied by the auto-labeller, not by a reporter. - Backport candidates for 1.0.x: triage of every PR merged since branch-1.0 was cut ([#5815](https://github.com/apache/datafusion-comet/issues/5815)) - A release-management triage record, described by its author as "a triage record, not a decision". It is neither a defect nor a feature request, and it was skipped on the same grounds in the 2026-09-14 pass. - Bug triage results: 2026-08-31 ([#5601](https://github.com/apache/datafusion-comet/issues/5601)) - Prior summary issue, open for three weeks. - Bug triage results: 2026-08-24 ([#5454](https://github.com/apache/datafusion-comet/issues/5454)) - Prior summary issue. Note that this one reports that **no labels were applied** in that pass because the token lacked Issues write permission, so its recommendations may still be unapplied; it needs more than a sign-off. - Bug triage results: 2026-08-17 ([#5385](https://github.com/apache/datafusion-comet/issues/5385)) - Prior summary issue, open for five weeks. - Bug triage results: 2026-08-11 ([#5325](https://github.com/apache/datafusion-comet/issues/5325)) - Prior summary issue, open for six weeks. - Bug triage results: 2026-08-03 ([#5231](https://github.com/apache/datafusion-comet/issues/5231)) - Prior summary issue, open for seven weeks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
