andygrove opened a new issue, #5601: URL: https://github.com/apache/datafusion-comet/issues/5601
Triage pass over the open `requires-triage` queue, per the project [Bug Triage Guide](https://github.com/apache/datafusion-comet/blob/main/docs/source/contributor-guide/bug_triage.md). - Date: 2026-08-31 - Total issues processed: 77 (73 triaged, 4 skipped, 0 failed) - Type counts: 24 bugs, 49 enhancements - Priority counts applied: `priority:critical` 5, `priority:high` 2, `priority:medium` 13, `priority:low` 4 - Guide: [docs/source/contributor-guide/bug_triage.md](https://github.com/apache/datafusion-comet/blob/main/docs/source/contributor-guide/bug_triage.md) Labels have already been applied. A reviewer should spot-check the calls below and close this issue when satisfied; corrections should be made directly on the affected issue. Note: where an issue already carried a priority label applied by its author, that label was left in place. Per the guide, this process adds a priority label to bugs only. ## Bugs ### priority:critical - Checked TIMESTAMP_MILLIS overflow for nested fields and nested-predicate scans is blocked on DataFusion nested-field pruning ([#5553](https://github.com/apache/datafusion-comet/issues/5553)) - Area labels: `area:scan` - Rationale: nested-field and nested-predicate scans keep overflow-to-NULL where Spark's `Math.multiplyExact` throws, which is a silent wrong result under decision-tree step 1, even though the fix is blocked on upstream nested-field pruning. - Match Spark ObjectHashAggregate decimal AVG buffer semantics ([#5509](https://github.com/apache/datafusion-comet/issues/5509)) - Area labels: `area:aggregation` - Rationale: grouped high-precision decimal AVG returns a different value from Spark with no error when Spark uses `ObjectHashAggregateExec`, a silent wrong result. - Match Spark ordering and rank semantics for floating values nested in arrays and structs ([#5507](https://github.com/apache/datafusion-comet/issues/5507)) - Area labels: `area:expressions` - Rationale: nested `-0.0`/NaN comparison keys give different `ORDER BY` and `RANK()` output from Spark with no error, a silent wrong result. - Support Spark-compatible Unicode case-insensitive Parquet field matching ([#5495](https://github.com/apache/datafusion-comet/issues/5495)) - Area labels: `area:scan` - Rationale: filed as a feature request, but the content is a correctness divergence — the native adapter's ASCII-only `eq_ignore_ascii_case` can miss a present physical column and return SQL NULL instead of the stored value under `spark.sql.caseSensitive=false`. - Date-to-timestamp casts can overflow or panic for wide dates ([#5456](https://github.com/apache/datafusion-comet/issues/5456)) - Area labels: `area:expressions` - Rationale: unchecked multiplication wraps to a bogus timestamp in release builds where Spark throws, which is a silent wrong result; the second reproducer additionally panics. ### priority:high - Native Azure store lets ambient AZURE_* environment variables override or corrupt explicit Hadoop auth config ([#5542](https://github.com/apache/datafusion-comet/issues/5542)) - Area labels: `area:scan` - Rationale: ambient env credentials silently win over explicitly configured Hadoop auth, so Comet and Spark can resolve different identities for the same table; major functional breakage with a security dimension. - Iceberg native scan claims schemes it cannot execute; three scheme lists disagree ([#5541](https://github.com/apache/datafusion-comet/issues/5541)) - Area labels: `area:scan` - Rationale: a `gcs`/`abfs`/`abfss`/`wasb`/`wasbs` Iceberg table passes validation, is claimed, and then every task dies with `CometNativeException`, while stock Spark reads it fine — major functional breakage on supported storage. ### priority:medium - Fix remaining reported issues for arrays of map ([#5544](https://github.com/apache/datafusion-comet/issues/5544)) - Area labels: `area:expressions` - Rationale: post-merge review of #5452 (merged) finds newly admitted complex literals and a `deepNullable` cast that changes Slice's nested map type while Slice still declares the original element type; a functional bug with a fallback workaround. - Iceberg native scan fails queries on tables whose column names are case-distinct to Java but not to Rust ([#5540](https://github.com/apache/datafusion-comet/issues/5540)) - Area labels: `area:scan` - Rationale: Rust's newer Unicode data folds columns Java keeps distinct, so the query dies with `_LEGACY_ERROR_TEMP_2093` — visible breakage with a fallback workaround. - Native Celeborn shuffle: validate_remote_schema rejects dictionary shapes the shuffle writer can emit ([#5536](https://github.com/apache/datafusion-comet/issues/5536)) - Area labels: `area:shuffle` - Rationale: local and remote readers disagree about valid frames, so a frame that reads locally fails on Celeborn; a broken feature, and the path is not yet enabled end to end. - NativeUtil.getNextBatch leaks Arrow structs when importVector fails ([#5534](https://github.com/apache/datafusion-comet/issues/5534)) - Area labels: `area:ffi`, `area:shuffle` - Rationale: the third exit from `getNextBatch` is unguarded, leaking C data and wrapper buffers on every native operator path; resource exhaustion rather than wrong results or an immediate crash. - unbase64 can fail on rows skipped by LIMIT and semi/anti joins ([#5532](https://github.com/apache/datafusion-comet/issues/5532)) - Area labels: `area:expressions` - Rationale: batch-at-a-time evaluation turns a query Spark completes into a decode failure; a visible functional bug with an expression-level opt-out. - Native Celeborn shuffle: the installed Celeborn bootstrap hook can break client creation for the whole executor ([#5529](https://github.com/apache/datafusion-comet/issues/5529)) - Area labels: `area:shuffle` - Rationale: a Comet-specific bootstrap failure is fatal to all Celeborn client creation on the executor; scoped medium because the native Celeborn path is not enabled end to end yet (see escalations). - Native Celeborn shuffle: reflectively replacing Celeborn's final fields can release push admission while payloads are in flight ([#5528](https://github.com/apache/datafusion-comet/issues/5528)) - Area labels: `area:shuffle` - Rationale: reflective assignment to four `private final` Celeborn fields can release admission early; a broken feature on a path that is not yet enabled. - Native Celeborn shuffle: default maxFrameBytes of 64 MiB is unreachable, and a large row fails the whole job ([#5527](https://github.com/apache/datafusion-comet/issues/5527)) - Area labels: `area:shuffle` - Rationale: the two shipped defaults cannot both hold, and the consequence for a large row is a failed job rather than a slow one; configurable, so a workaround exists. - Comet native broadcast fails under spark.kryo.registrationRequired=true ([#5510](https://github.com/apache/datafusion-comet/issues/5510)) - Area labels: none - Rationale: `Array[ChunkedByteBuffer]` is unregistered so broadcast throws outright; broken feature, worked around by not setting that Kryo option. - Avoid object-store cache and registry collisions across backends and configurations ([#5502](https://github.com/apache/datafusion-comet/issues/5502)) - Area labels: `area:scan` - Rationale: the cache key omits the backend, so a second URL can be served the first URL's store and the DataFusion registry can replace a mapping; a functional bug reachable only with mixed `fs.comet.libhdfs.schemes` routing. - Large-offset Arrow vectors from PyArrow UDFs cannot be serialized for broadcast or collect ([#5488](https://github.com/apache/datafusion-comet/issues/5488)) - Area labels: `area:ffi` - Rationale: `Utils.getFieldVector` throws `Unsupported Arrow Vector for serialize` for representations Comet deliberately produces elsewhere; visible failure on a narrow path. - AQE + DPP + spark.comet.exec.transitionRevert.enabled fails with "SubqueryAdaptiveBroadcastExec does not support the execute() code path" ([#5486](https://github.com/apache/datafusion-comet/issues/5486)) - Area labels: none - Rationale: reversion leaves the plan in a state where `PlanAdaptiveDynamicPruningFilters` no longer matches, failing the query; gated on a non-default config, so a workaround exists. - CometExecRule overwrites direct AQE LogicalQueryStage links during replanning ([#5482](https://github.com/apache/datafusion-comet/issues/5482)) - Area labels: none - Rationale: unconditional restoration of `originalPlan.logicalLink` breaks the correspondence between the current physical root and the active logical stage; a planner defect with no reported wrong-result or crash path. ### priority:low - Surface Parquet TIMESTAMP_MILLIS overflow as a Spark-faithful exception instead of a raw Arrow error ([#5517](https://github.com/apache/datafusion-comet/issues/5517)) - Area labels: `area:scan` - Rationale: the value is correctly rejected, only the exception type and message diverge from Spark's `ArithmeticException("long overflow")`; error-fidelity, no wrong results. - Cancel background batch producers before collecting final plan metrics ([#5504](https://github.com/apache/datafusion-comet/issues/5504)) - Area labels: `area:ffi` - Rationale: a drop-time metric guard can update counters after the final snapshot, so the reported metrics are incomplete; observability only. - Explain ObjectHashAggregate fallback when Comet shuffle is disabled ([#5500](https://github.com/apache/datafusion-comet/issues/5500)) - Area labels: `area:aggregation` - Rationale: the shuffle guard bypasses `withFallbackReason`, so the strict check can report an unannotated decline; a diagnostics gap, not an execution defect. - Track provenance of fallback reasons copied through shared expression nodes ([#5499](https://github.com/apache/datafusion-comet/issues/5499)) - Area labels: `area:expressions` - Rationale: a stale `FALLBACK_REASONS` tag on a shared node can satisfy `reportUnexplainedFallback` and hide an unexplained fallback; diagnostics only. ## Enhancements - Derive a native UDF's return type from the library instead of requiring the caller to declare it ([#5597](https://github.com/apache/datafusion-comet/issues/5597)) - Area labels: `area:udf` (pre-existing) - Rationale: an API ergonomics improvement to the native UDF registration surface; nothing is broken today. - Align Comet's native UDF surface with SPARK-55278's language-agnostic UDF protocol ([#5596](https://github.com/apache/datafusion-comet/issues/5596)) - Area labels: `area:udf` (pre-existing) - Rationale: forward-looking alignment with an accepted SPIP targeting Spark 4.3/4.4. - Rename the .claude directory to a vendor-neutral .ai ([#5592](https://github.com/apache/datafusion-comet/issues/5592)) - Area labels: none - Rationale: repository layout change matching `apache/datafusion` and `apache/datafusion-python`; no functional impact. - `next_day` and `levenshtein` fall back to Spark on collated strings ([#5591](https://github.com/apache/datafusion-comet/issues/5591)) - Area labels: `area:expressions` - Rationale: part of #5572; the current fallback is correct, the request is to route it through the codegen dispatcher instead. - `map_sort` falls back to Spark for non-scalar map key types ([#5590](https://github.com/apache/datafusion-comet/issues/5590)) - Area labels: `area:expressions` - Rationale: part of #5572; widening dispatcher coverage, not a defect. - `map_from_arrays` falls back to Spark under `mapKeyDedupPolicy=LAST_WIN`, unlike `map_from_entries` ([#5589](https://github.com/apache/datafusion-comet/issues/5589)) - Area labels: `area:expressions` - Rationale: part of #5572; requests a `CodegenDispatchFallback` mixin for parity with a sibling serde. - `timestamp_seconds` falls back to Spark for decimal, byte and short input ([#5588](https://github.com/apache/datafusion-comet/issues/5588)) - Area labels: `area:expressions` - Rationale: part of #5572; new input-type coverage. - `abs` on interval types falls back to Spark ([#5587](https://github.com/apache/datafusion-comet/issues/5587)) - Area labels: `area:expressions` - Rationale: part of #5572; new input-type coverage, already flagged `good first issue`. - `named_struct` with duplicate field names falls back to Spark ([#5586](https://github.com/apache/datafusion-comet/issues/5586)) - Area labels: `area:expressions` - Rationale: part of #5572; documented fallback, request is to dispatch instead. - `translate` falls back to Spark by default instead of using the codegen dispatcher like the other string functions ([#5585](https://github.com/apache/datafusion-comet/issues/5585)) - Area labels: `area:expressions` - Rationale: part of #5572; the `Incompatible` marking is correct, the ask is dispatcher coverage. - `length` / `bit_length` / `octet_length` fall back to Spark on binary input ([#5584](https://github.com/apache/datafusion-comet/issues/5584)) - Area labels: `area:expressions` - Rationale: part of #5572; new input-type coverage. - `arrays_zip` falls back to Spark for map element types ([#5583](https://github.com/apache/datafusion-comet/issues/5583)) - Area labels: `area:expressions` - Rationale: part of #5572; new element-type coverage. - Array functions fall back to Spark for binary and struct element types (`ArraysBase` type gate) ([#5582](https://github.com/apache/datafusion-comet/issues/5582)) - Area labels: `area:expressions` - Rationale: part of #5572; widening a type gate across seven serdes. - Hash functions fall back to Spark for decimal precision > 18, and `sha2` for a non-literal `numBits` ([#5581](https://github.com/apache/datafusion-comet/issues/5581)) - Area labels: `area:expressions` - Rationale: part of #5572; a performance/coverage gap, the fallback itself is correct. - Map lookups with float, collated or complex keys fall back to Spark (`map_col[key]`, `element_at`) ([#5580](https://github.com/apache/datafusion-comet/issues/5580)) - Area labels: `area:expressions` - Rationale: part of #5572; the gate exists for real semantic reasons, the ask is to dispatch rather than fall back. - `lpad` / `rpad` with a non-literal `pad` argument falls back to Spark ([#5579](https://github.com/apache/datafusion-comet/issues/5579)) - Area labels: `area:expressions` - Rationale: part of #5572; coverage for an ordinary query shape. - `to_csv` never runs inside Comet by default, unlike `to_json` / `from_csv` / `schema_of_csv` ([#5578](https://github.com/apache/datafusion-comet/issues/5578)) - Area labels: `area:expressions` - Rationale: part of #5572; coverage plus a documentation update. - `unix_timestamp` on string input falls back to Spark, while `to_unix_timestamp` already uses the codegen dispatcher ([#5577](https://github.com/apache/datafusion-comet/issues/5577)) - Area labels: `area:expressions` - Rationale: part of #5572; parity with an equivalent serde. - `round` on float/double falls back to Spark, while `bround` already uses the codegen dispatcher ([#5576](https://github.com/apache/datafusion-comet/issues/5576)) - Area labels: `area:expressions` - Rationale: part of #5572; parity with an equivalent serde. - Route unrecognized `StaticInvoke` and `Invoke` through the codegen dispatcher instead of falling back ([#5575](https://github.com/apache/datafusion-comet/issues/5575)) - Area labels: `area:expressions` - Rationale: part of #5572; turns a hard fallback default into in-pipeline execution. - Codegen dispatcher is unreachable from `convert`, so serdes that decline there never get a dispatch attempt ([#5574](https://github.com/apache/datafusion-comet/issues/5574)) - Area labels: `area:expressions` - Rationale: part of #5572; a structural gap in dispatcher reach rather than incorrect output. - Codegen dispatcher: guard the closure-serialize step so a non-serializable tree falls back cleanly instead of throwing at plan time ([#5573](https://github.com/apache/datafusion-comet/issues/5573)) - Area labels: `area:expressions` - Rationale: part of #5572; hardening an unexercised failure mode with no reported occurrence. - [EPIC] Codegen-dispatch coverage audit: expressions that fall back to Spark where the JVM dispatcher would work ([#5572](https://github.com/apache/datafusion-comet/issues/5572)) - Area labels: `area:expressions` - Rationale: umbrella for the coverage work above; a performance and coverage effort. - Narrow the invalid-UTF-8 Comet opt-out in the sketch and hll SQL test files ([#5571](https://github.com/apache/datafusion-comet/issues/5571)) - Area labels: `area:scan` - Rationale: the opt-out is correct but coarser than the problem; test-coverage improvement. - SQLQueryTestSuite.ignoreList entries skip the Spark baseline, not just Comet ([#5570](https://github.com/apache/datafusion-comet/issues/5570)) - Area labels: `area:ci` - Rationale: an intentional exclusion that is coarser than intended; narrowing it restores coverage rather than fixing a failure. - Remove stale dev/diffs test exclusions whose tracking issues are closed ([#5569](https://github.com/apache/datafusion-comet/issues/5569)) - Area labels: `area:ci` - Rationale: cleanup of exclusions whose cited issues are fixed; restores coverage. - Native shuffle rejects nested types as hash partitioning keys although the native hasher supports them ([#5566](https://github.com/apache/datafusion-comet/issues/5566)) - Area labels: `area:shuffle` - Rationale: the current gate is conservative but safe; the ask is to widen it now that nested hashing exists. - Support useLargeVarTypes in accelerated mapInArrow/mapInPandas ([#5555](https://github.com/apache/datafusion-comet/issues/5555)) - Area labels: none - Rationale: new configuration coverage for the accelerated Python UDF path. - [FEATURE] Enable direct top-level Variant projection in ordinary native Parquet scans ([#5551](https://github.com/apache/datafusion-comet/issues/5551)) - Area labels: `area:scan` - Rationale: new functionality, one of five atomic PRs under #5546. - [FEATURE] Match Spark physical Parquet semantics for projected Variant columns ([#5550](https://github.com/apache/datafusion-comet/issues/5550)) - Area labels: `area:scan` - Rationale: new functionality under #5546; the Variant path it constrains is not yet shipped. - [FEATURE] Normalize marked Variant arrays at the native Parquet boundary ([#5549](https://github.com/apache/datafusion-comet/issues/5549)) - Area labels: `area:scan` - Rationale: new functionality under #5546. - [FEATURE] Carry Spark VariantType identity through Comet schema serialization ([#5548](https://github.com/apache/datafusion-comet/issues/5548)) - Area labels: `area:scan`, `area:ffi` - Rationale: new protobuf and Arrow Field plumbing under #5546. - [FEATURE] Export complete Arrow Fields through the native FFI boundary ([#5547](https://github.com/apache/datafusion-comet/issues/5547)) - Area labels: `area:ffi` - Rationale: new FFI capability under #5546; today's exporter is limited, not broken. - [FEATURE] Native top-level Variant projection from ordinary Parquet ([#5546](https://github.com/apache/datafusion-comet/issues/5546)) - Area labels: `area:scan` - Rationale: parent tracking issue for re-landing #5407 as five reviewable PRs. - Native Celeborn shuffle: the remote read path costs up to 4.5x local decode and is unbenchmarked ([#5535](https://github.com/apache/datafusion-comet/issues/5535)) - Area labels: `area:shuffle` - Rationale: validation is the right call for untrusted bytes; the ask is measurement and optimization. - Native Celeborn shuffle: add a Celeborn test dependency so the reflection against its internals is verified ([#5530](https://github.com/apache/datafusion-comet/issues/5530)) - Area labels: `area:shuffle` - Rationale: test-infrastructure improvement. - Support NullType output types in codegen dispatch ([#5525](https://github.com/apache/datafusion-comet/issues/5525)) - Area labels: `area:expressions` - Rationale: widening the dispatch type gate for common untyped literals; the current rejection is safe. - Support Spark's pushed one-field VariantStruct (whole-value pushVariantIntoScan rewrite) in native Parquet scans ([#5519](https://github.com/apache/datafusion-comet/issues/5519)) - Area labels: `area:scan` - Rationale: new scan capability; the current behavior is an explicit Spark fallback. - Expose native Parquet scan I/O and read-amplification metrics ([#5508](https://github.com/apache/datafusion-comet/issues/5508)) - Area labels: `area:scan` - Rationale: new observability counters. - Narrow strict floating-point admission for corrected scalar sort keys ([#5506](https://github.com/apache/datafusion-comet/issues/5506)) - Area labels: `area:expressions` - Rationale: admission is conservative but correct after #5469; the ask is to relax it for scalar keys. - Include days in CalendarInterval hash when Spark 4.3+ is supported (SPARK-58236) ([#5498](https://github.com/apache/datafusion-comet/issues/5498)) - Area labels: `area:expressions` - Rationale: Comet matches every currently supported Spark version; this is prospective work for a Spark 4.3 profile. - ci: Prebuild Linux CI images with toolchains and dependencies ([#5490](https://github.com/apache/datafusion-comet/issues/5490)) - Area labels: `area:ci` - Rationale: build-infrastructure improvement for reliability and setup time. - Adopt techniques from Spark's ArrowCachedBatchSerializer (SPARK-57268) in Comet's cache format ([#5487](https://github.com/apache/datafusion-comet/issues/5487)) - Area labels: none - Rationale: records design ideas worth adopting from an upstream implementation of the same idea. - Cached reads that feed Spark operators are slower than Spark's own cache format ([#5485](https://github.com/apache/datafusion-comet/issues/5485)) - Area labels: none - Rationale: a performance gap in a new opt-in feature, not a regression against previously shipped behavior. - Comet cache decode ignores column projection, making narrow reads of wide cached relations slower than Spark ([#5484](https://github.com/apache/datafusion-comet/issues/5484)) - Area labels: none - Rationale: a design limitation of the new cache serializer from #5051; optimization request. - Track and remove native Variant compatibility workarounds after upstream fixes ([#5477](https://github.com/apache/datafusion-comet/issues/5477)) - Area labels: `area:scan` - Rationale: maintenance bookkeeping so intentionally narrow workarounds do not become permanent. - Remove Variant UTF-16 output rewriting ([#5474](https://github.com/apache/datafusion-comet/issues/5474)) - Area labels: `area:scan` - Rationale: removal of a correctness-preserving workaround once upstream ordering lands; current behavior is correct. - Make fair_unified account for spillable consumers ([#5465](https://github.com/apache/datafusion-comet/issues/5465)) - Area labels: none - Rationale: memory-pool accounting improvement matching DataFusion's `FairSpillPool`; deliberately left out of the focused #5212 fix. - Track exact SpillWriter page bytes for O(1) memory usage sampling ([#5462](https://github.com/apache/datafusion-comet/issues/5462)) - Area labels: `area:shuffle` - Rationale: performance follow-up to #5212 finding #12; the correctness fix stays separately scoped. ## Escalations to consider - Native Azure store lets ambient AZURE_* environment variables override or corrupt explicit Hadoop auth config ([#5542](https://github.com/apache/datafusion-comet/issues/5542)) - The guide lists security vulnerabilities under `priority:critical`. Ambient environment variables silently overriding an explicitly configured service principal (including the AKS workload-identity webhook case, where the configured client secret is dropped) is an identity-selection issue with a security dimension. Filed `priority:high`; a reviewer may want to escalate. - Date-to-timestamp casts can overflow or panic for wide dates ([#5456](https://github.com/apache/datafusion-comet/issues/5456)) - Matches the guide's escalation trigger in reverse: it both panics (`NaiveDate + TimeDelta overflowed`) and, in release builds, silently wraps to a wrong timestamp. Filed `priority:critical` on the wrong-result path. - Native Celeborn shuffle: the installed Celeborn bootstrap hook can break client creation for the whole executor ([#5529](https://github.com/apache/datafusion-comet/issues/5529)) - A Comet-specific bootstrap failure is fatal to all Celeborn shuffle on the executor, not just Comet's. Held at `priority:medium` because the native Celeborn path is not enabled end to end yet (see #5535); escalate to `priority:high` once it ships enabled. - Checked TIMESTAMP_MILLIS overflow for nested fields and nested-predicate scans is blocked on DataFusion nested-field pruning ([#5553](https://github.com/apache/datafusion-comet/issues/5553)) - Filed `priority:critical` on the silent-wrong-result rule, but the fix is blocked on upstream DataFusion nested-field pruning, so the priority reflects impact rather than actionability. ## Skipped — needs more info - Bug triage results: 2026-08-24 ([#5454](https://github.com/apache/datafusion-comet/issues/5454)) - A prior triage summary, not a bug or an enhancement, so no type label applies. `requires-triage` left in place. Note that pass applied no labels (its token lacked label write access), so its 28 issues were re-triaged in this pass. A reviewer should close it. - Bug triage results: 2026-08-17 ([#5385](https://github.com/apache/datafusion-comet/issues/5385)) - A prior triage summary awaiting reviewer sign-off; no type label applies and `requires-triage` was left in place. - Bug triage results: 2026-08-11 ([#5325](https://github.com/apache/datafusion-comet/issues/5325)) - A prior triage summary awaiting reviewer sign-off; no type label applies and `requires-triage` was left in place. - Bug triage results: 2026-08-03 ([#5231](https://github.com/apache/datafusion-comet/issues/5231)) - A prior triage summary awaiting reviewer sign-off; no type label applies and `requires-triage` was left in place. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
