andygrove opened a new issue, #7105: URL: https://github.com/apache/datafusion-comet/issues/7105
Triage pass over the open `requires-triage` queue, per the project [Bug Triage Guide](https://github.com/apache/datafusion-comet/blob/main/docs/source/contributor-guide/bug_triage.md). - Date: 2026-10-18 - Total issues processed: 107 (100 triaged, 7 skipped, 0 failed) - Type counts: 30 bugs, 70 enhancements - Priority counts applied: `priority:critical` 6, `priority:high` 2, `priority:medium` 14, `priority:low` 8 - Guide: [docs/source/contributor-guide/bug_triage.md](https://github.com/apache/datafusion-comet/blob/main/docs/source/contributor-guide/bug_triage.md) Labels have already been applied. A reviewer should spot-check the calls below and close this issue when satisfied. Corrections should be made directly on the affected issue. Notes on this pass: - **Most issues arrived typed.** 87 of the 100 triaged issues already carried `bug` or `enhancement`, and 15 of the bugs already carried a priority. Those labels were kept when they match the guide, and the rationale says "Confirms the author's labels". The 13 that arrived with no type were classified from their bodies: #7102, #7101, #7086, #7083, #7003, #7002, #7001, #6971, #6970, #6967, #6966, #6931 and #6911. - **Two type corrections.** #6934 carried both `bug` and `enhancement`, and #6997 carried `bug` for a pure refactor. Both are now `enhancement`. Either can be flipped back with one edit. - **Eight author-typed bugs arrived without a priority** and received one here: #7092, #7088, #7041, #7006, #6994, #6933, #6922 and #6898. - The guide's area table still lacks `area:joins`, `area:Iceberg`, `area:udf`, `area:memory` and `area:Delta`, and the repository has `spark 3.x`, `spark 4.0`, `spark 4.1` and `spark 4.2` rather than `spark 4`, so this pass added none of them. Issues that would have taken `area:joins` and don't carry it: #6922 and #6898. `regression` and `correctness` are also not in the guide and were not added. - No enhancement in this queue carries a `priority:*` label. ## Bugs ### priority:critical - Native Parquet scan returns NULL for nested BINARY values that are not valid UTF-8 when they are read as STRING ([#7032](https://github.com/apache/datafusion-comet/issues/7032)) - Area labels: `area:scan`; also carries `correctness` - Rationale: Silent wrong results: nested BINARY that is not valid UTF-8 reads as NULL where Spark returns the value, step 1. Confirms the author's labels. - Spark 3.4: native Parquet scan returns NULL for a struct matched by field id under a new name when Spark prunes its fields ([#7031](https://github.com/apache/datafusion-comet/issues/7031)) - Area labels: `area:scan`; also carries `correctness`, `spark 3.x` - Rationale: Silent wrong results: a struct matched by field id under a new name reads NULL on Spark 3.4, step 1. Confirms the author's labels. - Native Parquet scan returns a non-null array or struct of NULLs where Spark returns NULL because no requested nested field is in the file ([#7030](https://github.com/apache/datafusion-comet/issues/7030)) - Area labels: `area:scan`; also carries `correctness` - Rationale: Silent wrong results: a non-null array or struct of NULLs comes back where Spark returns NULL, step 1. Confirms the author's labels. - Native Parquet scan returns wrong INT96 timestamps inside legacy-format lists ([#7029](https://github.com/apache/datafusion-comet/issues/7029)) - Area labels: `area:scan`; also carries `correctness` - Rationale: Silent wrong results: INT96 timestamps inside legacy-format lists come back wrong, step 1. Confirms the author's labels. - Native outer hash joins still lose a sorted streamed side's order over some caches and checkpoints ([#6922](https://github.com/apache/datafusion-comet/issues/6922)) - Area labels: none - Rationale: Rows come back in the wrong order and Spark has already dropped the sort that would fix them, so the result differs silently, step 1, the same symptom as #6787. It needs a cache or checkpoint shape that is narrower than #6787's, and `main` is no worse for it. - Native broadcast nested loop joins report the streamed side's order without keeping it ([#6898](https://github.com/apache/datafusion-comet/issues/6898)) - Area labels: none - Rationale: With default settings `sortWithinPartitions` returns unsorted partitions, and with AQE off a sort-merge join above the join returns wrong rows, so results differ silently, step 1. The same class as #6787 and #6922. ### priority:high - Native hashing of NullType and ANSI interval values fails with "Unsupported data type in hasher" ([#7063](https://github.com/apache/datafusion-comet/issues/7063)) - Area labels: `area:expressions`, `area:shuffle` - Rationale: Hashing a NullType or ANSI interval value fails the task natively, a visible failure rather than wrong results, step 2. Confirms the author's labels. - Negative-scale decimal cast to timestamp panics ([#7002](https://github.com/apache/datafusion-comet/issues/7002)) - Area labels: `area:expressions` - Rationale: A negative-scale decimal cast to timestamp panics in `10_i128.pow`, step 2. It needs `spark.sql.legacy.allowNegativeScaleOfDecimal=true`, but `CometCast.canCastFromDecimal` reports the cast as compatible for any scale. ### priority:medium - BoundedWindowAggExec reserves no memory for the rows its window frames keep ([#7092](https://github.com/apache/datafusion-comet/issues/7092)) - Area labels: none; also carries `area:memory` - Rationale: `BoundedWindowAggExec` keeps rows its frames need without reserving memory, so a skewed frame can grow past the container limit, step 3. It hasn't been measured, and the common frames are small. - fair_unified's per-consumer limit does not bind when tasks run concurrently, so one consumer can starve another ([#7088](https://github.com/apache/datafusion-comet/issues/7088)) - Area labels: none - Rationale: The fair pool's per-consumer limit doesn't bind with concurrent tasks, so one sort can starve another. Tasks fail but succeeded on retry in 38 of 38 attempts reported, step 3. - map_from_entries dispatches a calendar-interval result into the microsecond overflow ([#7083](https://github.com/apache/datafusion-comet/issues/7083)) - Area labels: `area:expressions` - Rationale: `map_from_entries` over calendar intervals longer than about 292 years aborts with `ArithmeticException` where Spark returns the map, a visible failure on an extreme input, step 3, tracked with #5279. - [EPIC] Nested types in the native Parquet and Iceberg writers: correctness and performance follow-ups ([#7051](https://github.com/apache/datafusion-comet/issues/7051)) - Area labels: `area:writer`; also carries `EPIC`, `area:Iceberg`, `correctness`, `performance` - Rationale: An EPIC of nested-type writer follow-ups, held at the level of its worst open item, as the author labeled it. Confirms the author's labels. - Native Parquet writer ignores the datetime rebase modes, so Spark rejects or shifts its pre-1582 dates ([#7047](https://github.com/apache/datafusion-comet/issues/7047)) - Area labels: `area:writer`; also carries `correctness` - Rationale: The native Parquet writer ignores the rebase modes, so Spark rejects or shifts its pre-1582 dates. The writer is still opt-in, so step 3 rather than step 1. Confirms the author's labels. - Native Iceberg scan fails when a data file omits an optional nested field ([#7033](https://github.com/apache/datafusion-comet/issues/7033)) - Area labels: `area:scan`; also carries `area:Iceberg` - Rationale: The native Iceberg scan fails where Spark reads a file that omits an optional nested field, a visible failure with a Spark fallback, step 3. Confirms the author's labels. - Native Parquet scan fails on a duplicate field id in a column the query does not read ([#7006](https://github.com/apache/datafusion-comet/issues/7006)) - Area labels: `area:scan` - Rationale: With `spark.sql.parquet.fieldId.read.enabled=true`, the native scan fails on a duplicate field id in a column the query never reads, where Spark reads the file. A visible failure behind an opt-in config, step 3. - Legacy cast of an overflowing finite double to timestamp returns NULL ([#7003](https://github.com/apache/datafusion-comet/issues/7003)) - Area labels: `area:expressions` - Rationale: A legacy double-to-timestamp cast returns NULL where Spark saturates, which is wrong output at step 1. It needs a value above about 9.2e12 seconds, so it is held at medium like the far-date bugs #5208 and #6145 (see escalations). - Native Iceberg scan fails a VERSION AS OF read of a column whose type was promoted after the snapshot ([#6993](https://github.com/apache/datafusion-comet/issues/6993)) - Area labels: `area:scan`; also carries `area:Iceberg` - Rationale: A `VERSION AS OF` read of a promoted column fails in the native Iceberg scan, a visible failure with a Spark fallback, step 3. Confirms the author's labels. - JVM codegen dispatch skips a shared subexpression on NULL rows under ANSI ([#6971](https://github.com/apache/datafusion-comet/issues/6971)) - Area labels: `area:expressions` - Rationale: Under ANSI with subexpression elimination, a dispatched expression skips a shared subexpression on NULL rows, so Comet returns rows where Spark fails the query. It needs ANSI on and a subexpression that raises, step 3. - Native element_at and map_from_arrays skip a shared subexpression on NULL rows under ANSI ([#6970](https://github.com/apache/datafusion-comet/issues/6970)) - Area labels: `area:expressions` - Rationale: The same ANSI shared-subexpression gap as #6971, through the native `element_at` and `map_from_arrays` NULL guards. Step 3. - Native string to decimal cast returns zero or NULL where Spark's rescale throws ArithmeticException ([#6966](https://github.com/apache/datafusion-comet/issues/6966)) - Area labels: `area:expressions` - Rationale: A native string-to-decimal cast returns `0.00` or NULL for exponents near 2^31 where Spark raises `ArithmeticException`. It needs an extreme exponent, so it is held at medium (see escalations). - Native Iceberg writes commit brotli files that no JVM reader can read without Hadoop's BrotliCodec ([#6960](https://github.com/apache/datafusion-comet/issues/6960)) - Area labels: `area:writer`; also carries `area:Iceberg` - Rationale: Files committed with brotli can't be read by a JVM reader without Hadoop's BrotliCodec, which breaks downstream readers of the table, step 3. Confirms the author's labels. - Native string and binary `min`/`max` under-report memory in hash aggregates ([#6936](https://github.com/apache/datafusion-comet/issues/6936)) - Area labels: `area:aggregation`; also carries `area:memory` - Rationale: Native string and binary `min`/`max` under-count memory in hash aggregates, so the pool can't refuse or spill in time, step 3. Confirms the author's labels. ### priority:low - Native Parquet writer ignores outputTimestampType and writeLegacyFormat, and omits Spark's schema metadata ([#7048](https://github.com/apache/datafusion-comet/issues/7048)) - Area labels: `area:writer` - Rationale: The writer ignores `outputTimestampType` and `writeLegacyFormat` and omits Spark's schema metadata, which affects file metadata more than values. Confirms the author's labels. - Native panic backtrace can be attached to another thread's CometNativeException ([#7041](https://github.com/apache/datafusion-comet/issues/7041)) - Area labels: `area:ffi` - Rationale: A native panic's backtrace can be attached to another thread's `CometNativeException`, which misleads debugging but doesn't change a result. Step 4. - Native Parquet scan cannot read a file whose schema has a nested encoding arrow-rs rejects, even in a column the query does not read ([#7035](https://github.com/apache/datafusion-comet/issues/7035)) - Area labels: `area:scan` - Rationale: The native scan can't read a file with a nested encoding arrow-rs rejects, even in a column the query doesn't read. It is a visible failure on a rare file layout. Confirms the author's labels. - Native Parquet scan reads TIMESTAMP(NANOS) columns as timestamp where Spark rejects them ([#7034](https://github.com/apache/datafusion-comet/issues/7034)) - Area labels: `area:scan` - Rationale: The native scan reads TIMESTAMP(NANOS) where Spark rejects it, so Comet is more permissive on a file Spark can't read. Confirms the author's labels. - Native Iceberg writes accept write.parquet.compression-codec=none, which iceberg-java rejects ([#7008](https://github.com/apache/datafusion-comet/issues/7008)) - Area labels: `area:writer`; also carries `area:Iceberg` - Rationale: The native Iceberg writer accepts `write.parquet.compression-codec=none`, which iceberg-java rejects, so a misconfigured table writes instead of failing. Confirms the author's labels. - ANSI NaN and infinity use the wrong exception for cast to timestamp ([#7001](https://github.com/apache/datafusion-comet/issues/7001)) - Area labels: `area:expressions` - Rationale: Under ANSI, a NaN or infinity cast to timestamp raises `SparkNumberFormatException` where Spark raises `SparkDateTimeException`. The error class and SQLSTATE match, and only the exception type and message text differ. Step 4. - Nested field-id reads give one file column to both an id match and a name match; the root returns null ([#6994](https://github.com/apache/datafusion-comet/issues/6994)) - Area labels: `area:scan` - Rationale: An opt-in field-id read where two requested fields select one file column gives null at the root and values in a struct. Spark's own readers disagree there (zeros, stale memory or null), so there is no stable answer to match. Step 4. - Broadcast serialization keeps 64-bit offsets for large_string and large_binary children of nested columns ([#6933](https://github.com/apache/datafusion-comet/issues/6933)) - Area labels: `area:ffi` - Rationale: Nested `large_string` and `large_binary` children keep 64-bit offsets in broadcast serialization. The report says it is latent on `main`, because no plan that reaches it exists yet. Step 4. ## Enhancements - External sort OOM error tells Comet users to change `datafusion.runtime.memory_limit` and `datafusion.execution.sort_spill_reservation_bytes` ([#7102](https://github.com/apache/datafusion-comet/issues/7102)) - Area labels: none - Rationale: An error message tells users to change configs that Comet doesn't expose, the same shape as #7091. - docs: update Scala comments that reference the removed iceberg-writes.md page ([#7101](https://github.com/apache/datafusion-comet/issues/7101)) - Area labels: none - Rationale: Updates four Scala comments that name a docs page #7090 removed. - Spill limit error tells Comet users to raise datafusion.runtime.max_temp_directory_size ([#7091](https://github.com/apache/datafusion-comet/issues/7091)) - Area labels: none - Rationale: An error message points users at a DataFusion config rather than a Comet one, a message improvement. Confirms the author's label. - Replace remaining unwrap calls on protobuf input in the native planner with errors ([#7086](https://github.com/apache/datafusion-comet/issues/7086)) - Area labels: none - Rationale: Hardening: the unwraps panic only on a protobuf shape the serde doesn't produce, and no query is reported to fail. - bench: add the shuffle benchmark cases that the #5905 review found missing ([#7080](https://github.com/apache/datafusion-comet/issues/7080)) - Area labels: `area:shuffle`; also carries `performance` - Rationale: Adds benchmark cases missing from #5905. Confirms the author's label. - perf: Celeborn push completion is polled every 10 ms and the transport hook proxies every channel of the shared factory ([#7079](https://github.com/apache/datafusion-comet/issues/7079)) - Area labels: `area:shuffle`; also carries `performance` - Rationale: A performance request for the Celeborn path. Confirms the author's label. - perf: the native Celeborn writer pays a large fixed cost per frame (admission, schema, estimates, re-encodes) ([#7078](https://github.com/apache/datafusion-comet/issues/7078)) - Area labels: `area:shuffle`; also carries `performance` - Rationale: A performance request for the native Celeborn writer's per-frame cost. Confirms the author's label. - perf: the native Celeborn writer sends one frame per partition chunk with no coalescing ([#7077](https://github.com/apache/datafusion-comet/issues/7077)) - Area labels: `area:shuffle`; also carries `performance` - Rationale: A performance request to coalesce Celeborn frames. Confirms the author's label. - perf: the native Celeborn writer reserves about 7x a batch and holds it through the network round trip ([#7076](https://github.com/apache/datafusion-comet/issues/7076)) - Area labels: `area:shuffle`; also carries `area:memory`, `performance` - Rationale: A request to reserve less memory in the Celeborn writer, with nothing failing. Confirms the author's label. - perf: the JVM shuffle's native radix sort always runs six passes, and the JVM reserves scratch it never uses ([#7075](https://github.com/apache/datafusion-comet/issues/7075)) - Area labels: `area:shuffle`; also carries `performance` - Rationale: A performance request for the JVM shuffle's radix sort. Confirms the author's label. - perf: the JVM columnar shuffle insert loop boxes, locks and copies every row more than once ([#7074](https://github.com/apache/datafusion-comet/issues/7074)) - Area labels: `area:shuffle`; also carries `performance` - Rationale: A performance request for the JVM columnar shuffle insert loop. Confirms the author's label. - perf: a shuffle over a native plan takes the JVM row shuffle for round-robin and nested hash partitioning ([#7073](https://github.com/apache/datafusion-comet/issues/7073)) - Area labels: `area:shuffle`; also carries `performance` - Rationale: A performance request to run more shuffles natively. Confirms the author's label. - perf: the native multi-partition shuffle writer copies each partition's tail batch and flushes the output file once per partition ([#7071](https://github.com/apache/datafusion-comet/issues/7071)) - Area labels: `area:shuffle`; also carries `performance` - Rationale: A performance request for the native multi-partition shuffle writer. Confirms the author's label. - perf: NativeBatchDecoderIterator discards its direct buffer after every block over 128 KiB ([#7070](https://github.com/apache/datafusion-comet/issues/7070)) - Area labels: `area:shuffle`; also carries `performance` - Rationale: A performance request for the shuffle block decoder. Confirms the author's label. - perf: investigate decoding shuffle blocks ahead of the consumer in native shuffle read ([#7069](https://github.com/apache/datafusion-comet/issues/7069)) - Area labels: `area:shuffle`; also carries `performance` - Rationale: An investigation into decoding shuffle blocks ahead of the consumer, a performance idea. Confirms the author's label. - perf: a stage that reads shuffle and ends in columnar-to-row crosses the FFI boundary twice per batch ([#7068](https://github.com/apache/datafusion-comet/issues/7068)) - Area labels: `area:ffi`, `area:shuffle`; also carries `performance` - Rationale: A performance request to avoid a double FFI crossing per batch. Confirms the author's label. - perf: the shuffle reader builds a new decompressor and zeroes a fresh body buffer for every block ([#7067](https://github.com/apache/datafusion-comet/issues/7067)) - Area labels: `area:shuffle`; also carries `performance` - Rationale: A performance request to reuse the shuffle reader's decompressor and buffer. Confirms the author's label. - perf: Arrow IPC write paths still copy through Channels.newChannel's 8 KiB buffer ([#7064](https://github.com/apache/datafusion-comet/issues/7064)) - Area labels: none; also carries `performance` - Rationale: A performance request to skip an 8 KiB channel buffer on Arrow IPC writes. Confirms the author's label. - perf: the sort-based JVM columnar shuffle is 2x slower than Spark at 2,000 partitions because every partition write pays builder, file and codec setup ([#7061](https://github.com/apache/datafusion-comet/issues/7061)) - Area labels: `area:shuffle`; also carries `performance` - Rationale: A performance gap against Spark at 2,000 partitions, with correct results. Confirms the author's label. - perf: every native plan creates and deletes a temp directory on each Spark local disk, even when it never spills ([#7060](https://github.com/apache/datafusion-comet/issues/7060)) - Area labels: none; also carries `performance` - Rationale: A performance request to skip the temp directory when a plan never spills. Confirms the author's label. - Native Iceberg writes into NOT NULL columns fall back on Spark 3.5+ because Comet has no native AssertNotNull ([#7049](https://github.com/apache/datafusion-comet/issues/7049)) - Area labels: `area:writer`; also carries `area:Iceberg`, `performance` - Rationale: Native support for `AssertNotNull` would keep Iceberg writes into NOT NULL columns native, and they are correct today as a fallback. Confirms the author's label. - Native Iceberg scan reads every column of a file when it needs none of them ([#7039](https://github.com/apache/datafusion-comet/issues/7039)) - Area labels: `area:scan`; also carries `area:Iceberg`, `performance` - Rationale: A performance request to read no columns when none are needed. Confirms the author's label. - Push Iceberg residual predicates on struct fields into the native scan ([#7038](https://github.com/apache/datafusion-comet/issues/7038)) - Area labels: `area:scan`; also carries `area:Iceberg`, `performance` - Rationale: A performance request to push struct-field predicates into the native Iceberg scan. Confirms the author's label. - Native Parquet scan does not prune row groups or pages with predicates on struct fields ([#7037](https://github.com/apache/datafusion-comet/issues/7037)) - Area labels: `area:scan`; also carries `performance` - Rationale: A performance request to prune row groups and pages on struct fields. Results are correct today. Confirms the author's label. - Native Parquet scan reads the whole nested column for maps, field ids, case-insensitive names and leaf type changes ([#7036](https://github.com/apache/datafusion-comet/issues/7036)) - Area labels: `area:scan`; also carries `performance` - Rationale: A performance request to read less of a nested column. Confirms the author's label. - Negative-scale decimal `/`, `div`, `ceil` and `floor` fall back to Spark while `round` dispatches ([#7023](https://github.com/apache/datafusion-comet/issues/7023)) - Area labels: `area:expressions` - Rationale: Four negative-scale decimal functions fall back to Spark, which gives correct results, so native support is new functionality. Confirms the author's label. - Native decimal cast kernels compute `10^scale` without checking for a negative scale ([#7022](https://github.com/apache/datafusion-comet/issues/7022)) - Area labels: `area:expressions`; also carries `rust`, `tech debt` - Rationale: Adds a negative-scale check to the native decimal kernels as tech debt. Confirms the author's label. - Use one shared check and reason string for negative-scale decimals across the serdes ([#7021](https://github.com/apache/datafusion-comet/issues/7021)) - Area labels: `area:expressions`; also carries `tech debt` - Rationale: A refactor to share one check and reason string across the serdes. Confirms the author's label. - Native Iceberg scan reads its full schema when a partition source is nested in a struct the query prunes ([#7019](https://github.com/apache/datafusion-comet/issues/7019)) - Area labels: none; also carries `area:Iceberg`, `performance` - Rationale: A performance request to prune a partition source nested in a struct. Confirms the author's label. - Lower and upper window-frame bound serialization are near-identical copies ([#6999](https://github.com/apache/datafusion-comet/issues/6999)) - Area labels: none; also carries `tech debt` - Rationale: A refactor of two near-identical window-frame serializers. Confirms the author's label. - Howard Hinnant civil-date algorithms hand-written 7 times across 3 crates ([#6997](https://github.com/apache/datafusion-comet/issues/6997)) - Area labels: none; also carries `tech debt` - Rationale: A refactor to share the civil-date algorithms copied across three crates. It arrived as `bug` from the bug template, but nothing is wrong with the results. - Reuse hash_list_primitive! for string/binary list elements and factor out map-entry hashing loops ([#6984](https://github.com/apache/datafusion-comet/issues/6984)) - Area labels: `area:expressions`; also carries `rust`, `tech debt` - Rationale: A refactor to reuse a hashing macro. Confirms the author's label. - Extract shared list-array construction from the split_* functions in split.rs ([#6983](https://github.com/apache/datafusion-comet/issues/6983)) - Area labels: `area:expressions`; also carries `rust`, `tech debt` - Rationale: A refactor of the `split_*` list construction. Confirms the author's label. - Replace duplicated scalar/array broadcast match in math_funcs with a shared helper ([#6982](https://github.com/apache/datafusion-comet/issues/6982)) - Area labels: `area:expressions`; also carries `rust`, `tech debt` - Rationale: A refactor of a duplicated broadcast match in `math_funcs`. Confirms the author's label. - Extract a variable-length slot writer from TypedElements::write_elements_slow in columnar_to_row.rs ([#6981](https://github.com/apache/datafusion-comet/issues/6981)) - Area labels: none; also carries `rust`, `tech debt` - Rationale: A refactor of a variable-length slot writer. Confirms the author's label. - Deduplicate ASCII and char paths in levenshtein_distance_with_threshold ([#6980](https://github.com/apache/datafusion-comet/issues/6980)) - Area labels: `area:expressions`; also carries `rust`, `tech debt` - Rationale: A refactor of `levenshtein_distance_with_threshold`. Confirms the author's label. - [EPIC] Finish moving PhysicalPlanner::create_plan and create_expr onto the operator and expression registries ([#6979](https://github.com/apache/datafusion-comet/issues/6979)) - Area labels: none; also carries `EPIC`, `rust`, `tech debt` - Rationale: An EPIC of planner refactors. Confirms the author's label. - Share Parquet field-ID and name resolution between remap_physical_schema and match_struct_fields ([#6978](https://github.com/apache/datafusion-comet/issues/6978)) - Area labels: `area:scan`; also carries `rust`, `tech debt` - Rationale: A refactor to share Parquet field-id and name resolution. Confirms the author's label. - Deduplicate field-major struct append paths in shuffle spark_unsafe/row.rs ([#6977](https://github.com/apache/datafusion-comet/issues/6977)) - Area labels: `area:shuffle`; also carries `rust`, `tech debt` - Rationale: A refactor of the struct append paths in shuffle `row.rs`. Confirms the author's label. - Deduplicate Legacy/Ansi/Try integer SUM accumulators in sum_int.rs ([#6976](https://github.com/apache/datafusion-comet/issues/6976)) - Area labels: `area:aggregation`; also carries `rust`, `tech debt` - Rationale: A refactor of the integer SUM accumulators. Confirms the author's label. - Extract a shared metrics-preserving exec wrapper from CometFilterExec and CometProjectionExec ([#6975](https://github.com/apache/datafusion-comet/issues/6975)) - Area labels: none; also carries `rust`, `tech debt` - Rationale: A refactor to share a metrics-preserving wrapper. Confirms the author's label. - Deduplicate spark_ceil and spark_floor and port floor's decimal boundary tests to ceil ([#6974](https://github.com/apache/datafusion-comet/issues/6974)) - Area labels: `area:expressions`; also carries `rust`, `tech debt` - Rationale: A refactor of `spark_ceil` and `spark_floor`, with test porting. Confirms the author's label. - Pass a scan's Hadoop options to libhdfs ([#6967](https://github.com/apache/datafusion-comet/issues/6967)) - Area labels: `area:scan` - Rationale: Passing a scan's Hadoop options to libhdfs is new functionality: the issue is filed from the feature request template, and a filesystem whose settings come only from those options fails natively today. - Native Iceberg writes: support `uuid` and `unknown` columns ([#6959](https://github.com/apache/datafusion-comet/issues/6959)) - Area labels: `area:writer`; also carries `area:Iceberg` - Rationale: Native writes of `uuid` and `unknown` columns would be new functionality, since these writes fall back today. Confirms the author's label. - Native Iceberg writes: support Azure and `oss` data locations, S3-compatible scheme aliases, and `gs` through `HadoopFileIO` ([#6958](https://github.com/apache/datafusion-comet/issues/6958)) - Area labels: `area:writer`; also carries `area:Iceberg` - Rationale: Native writes to Azure, `oss` and `gs` locations would be new functionality. Confirms the author's label. - Native Iceberg writes: admit S3 settings that cannot change the written object, and translate S3 credential providers and encryption settings ([#6957](https://github.com/apache/datafusion-comet/issues/6957)) - Area labels: `area:writer`; also carries `area:Iceberg` - Rationale: Admitting more S3 settings in native Iceberg writes is new functionality. Confirms the author's label. - Native Iceberg writes: support the object-storage layout (`write.object-storage.enabled=true`) ([#6956](https://github.com/apache/datafusion-comet/issues/6956)) - Area labels: `area:writer`; also carries `area:Iceberg` - Rationale: Support for the object-storage layout is new functionality. Confirms the author's label. - Native Iceberg writes: stop falling back on settings iceberg-java's writer ignores ([#6955](https://github.com/apache/datafusion-comet/issues/6955)) - Area labels: `area:writer`; also carries `area:Iceberg` - Rationale: Fewer fallbacks for settings iceberg-java ignores is new functionality. Confirms the author's label. - Native Iceberg writes: translate the Parquet settings that have a parquet-rs equivalent ([#6954](https://github.com/apache/datafusion-comet/issues/6954)) - Area labels: `area:writer`; also carries `area:Iceberg` - Rationale: Translating Parquet settings for the native writer is new functionality. Confirms the author's label. - On Spark 4.0+, an adaptive cached plan that ends in a Spark row operator is converted into Spark column vectors and then copied into Arrow ([#6952](https://github.com/apache/datafusion-comet/issues/6952)) - Area labels: `area:scan`; also carries `performance` - Rationale: A performance request to avoid a vector conversion and copy for cached plans. Confirms the author's label. - Every cached batch allocates a 1 MiB heap chunk, however small its payload ([#6951](https://github.com/apache/datafusion-comet/issues/6951)) - Area labels: `area:scan`; also carries `performance` - Rationale: A performance request to size the cache's heap chunk to the payload. Confirms the author's label. - The in-memory cache stores small input batches as they arrive, so a cached repartition over many map tasks holds thousands of small batches ([#6950](https://github.com/apache/datafusion-comet/issues/6950)) - Area labels: `area:scan`; also carries `performance` - Rationale: A performance request to coalesce small batches in the in-memory cache. Confirms the author's label. - Upload S3A magic committer files natively instead of through the JVM ([#6944](https://github.com/apache/datafusion-comet/issues/6944)) - Area labels: `area:writer` - Rationale: Uploading magic committer files natively is new functionality. Confirms the author's label. - Store offsets as lengths in shuffle blocks to shrink shuffle files ([#6941](https://github.com/apache/datafusion-comet/issues/6941)) - Area labels: `area:shuffle`; also carries `performance` - Rationale: A request to shrink shuffle files, a performance and size optimization. Confirms the author's label. - Spark's task CPU time (`executorCpuTime`) leaves out native execution for plans with no JVM input ([#6935](https://github.com/apache/datafusion-comet/issues/6935)) - Area labels: none - Rationale: The task CPU metric leaves out native work, a metrics improvement with correct query results. Confirms the author's label. - Bump iceberg-rust past apache/iceberg-rust#3255 and narrow the nested schema evolution fallback ([#6934](https://github.com/apache/datafusion-comet/issues/6934)) - Area labels: `area:scan`; also carries `area:Iceberg` - Rationale: A pin bump that narrows a deliberate fallback, filed from the feature request template. The fallback keeps results correct today, so nothing is broken. The issue arrived with both `bug` and `enhancement`. - Native Delta scan declines Hadoop's S3A ProfileAWSCredentialsProvider ([#6931](https://github.com/apache/datafusion-comet/issues/6931)) - Area labels: `area:scan` - Rationale: Adding support for Hadoop's `ProfileAWSCredentialsProvider` to the native Delta scan, which declines it and falls back today. - Support the S3A magic committer in native Parquet writes to S3 ([#6929](https://github.com/apache/datafusion-comet/issues/6929)) - Area labels: `area:writer` - Rationale: Support for the S3A magic committer in native writes is new functionality. Confirms the author's label. - Faster nested Parquet decoding in the native Iceberg scan (parquet-rs) ([#6921](https://github.com/apache/datafusion-comet/issues/6921)) - Area labels: `area:scan`; also carries `area:Iceberg`, `performance` - Rationale: A performance request for nested Parquet decoding. Confirms the author's label. - Reduce string dictionary interning and level computation cost in the parquet-rs writer ([#6920](https://github.com/apache/datafusion-comet/issues/6920)) - Area labels: `area:writer`; also carries `performance` - Rationale: A performance request for the Parquet writer. Confirms the author's label. - Faster string min/max statistics in the parquet-rs writer ([#6919](https://github.com/apache/datafusion-comet/issues/6919)) - Area labels: `area:writer`; also carries `performance` - Rationale: A performance request for string min/max statistics in the writer. Confirms the author's label. - Faster Parquet level encoding for nested columns (parquet-rs RleEncoder) ([#6918](https://github.com/apache/datafusion-comet/issues/6918)) - Area labels: `area:writer`; also carries `performance` - Rationale: A performance request for Parquet level encoding. Confirms the author's label. - Reduce the cost of LZ4 compression in native shuffle ([#6916](https://github.com/apache/datafusion-comet/issues/6916)) - Area labels: `area:shuffle`; also carries `performance` - Rationale: A performance request for LZ4 cost in native shuffle. Confirms the author's label. - Native Iceberg writes do not use shuffle direct read for their input ([#6913](https://github.com/apache/datafusion-comet/issues/6913)) - Area labels: `area:shuffle`, `area:writer`; also carries `area:Iceberg`, `performance` - Rationale: A performance request to use shuffle direct read in native Iceberg writes. Confirms the author's label. - [EPIC] Performance follow-ups from profiling the end-to-end Iceberg benchmark ([#6911](https://github.com/apache/datafusion-comet/issues/6911)) - Area labels: none; also carries `EPIC`, `area:Iceberg`, `performance` - Rationale: A performance EPIC from profiling the Iceberg benchmark, which shows Comet 2.0x to 2.9x faster than Spark. It arrived with no type label. - Native filter copies its surviving rows, which makes filter-then-aggregate on narrow tables slower than Spark ([#6908](https://github.com/apache/datafusion-comet/issues/6908)) - Area labels: none; also carries `performance` - Rationale: A performance gap against Spark on narrow tables, with correct results. Confirms the author's label. - canAggregateBeConverted does not mirror doConvert's collated grouping key guard ([#6906](https://github.com/apache/datafusion-comet/issues/6906)) - Area labels: `area:aggregation` - Rationale: A planning-time check that doesn't mirror the conversion guard, so a fallback decision is made late. Confirms the author's label. - Machine-readable report of Comet coverage and fallback reasons ([#6904](https://github.com/apache/datafusion-comet/issues/6904)) - Area labels: none - Rationale: A machine-readable coverage report is new functionality. Confirms the author's label. - Joins with join type `Cross` (`CROSS JOIN`, `Dataset.crossJoin`) fall back to Spark ([#6903](https://github.com/apache/datafusion-comet/issues/6903)) - Area labels: none - Rationale: Native `CROSS JOIN` is new functionality, since it falls back to Spark with correct results. Confirms the author's label. - Behavior-flag audit: fix Spark version comments on native flags and centralize the CollectSet check ([#6893](https://github.com/apache/datafusion-comet/issues/6893)) - Area labels: `area:aggregation`, `area:scan` - Rationale: Comment fixes and a centralized check from a behavior-flag audit. Confirms the author's label. ## Escalations to consider - Native outer hash joins still lose a sorted streamed side's order over some caches and checkpoints ([#6922](https://github.com/apache/datafusion-comet/issues/6922)) - Labeled critical for the lost sort, the symptom of #6787. Both shapes behave as on `main`, and they need a cache over an outer join, or a `localCheckpoint`, so a reviewer might prefer `priority:high` or medium for a narrower reach. - Native broadcast nested loop joins report the streamed side's order without keeping it ([#6898](https://github.com/apache/datafusion-comet/issues/6898)) - Labeled critical because default settings return unsorted partitions after `sortWithinPartitions`, and with AQE off a sort-merge join above it returns wrong rows. It needs a broadcast nested loop join over sorted input, which is an uncommon plan. - Legacy cast of an overflowing finite double to timestamp returns NULL ([#7003](https://github.com/apache/datafusion-comet/issues/7003)) - Held at medium, with #6966, following #5208 and #6145 for values real data doesn't reach. Step 1 reads critical for any default-config difference from Spark. The reviewer may prefer critical, as for #6839. - Native string to decimal cast returns zero or NULL where Spark's rescale throws ArithmeticException ([#6966](https://github.com/apache/datafusion-comet/issues/6966)) - Held at medium because it needs an exponent near 2^31, such as `'1e-2147483647'`. The same class of string-to-decimal difference, #6839, is critical, so the reviewer may want to compare. - Negative-scale decimal cast to timestamp panics ([#7002](https://github.com/apache/datafusion-comet/issues/7002)) - Labeled high for the panic, but it needs `spark.sql.legacy.allowNegativeScaleOfDecimal=true`. The reviewer may prefer medium on that opt-in, and note #7022 and #7023 for the negative-scale decimal cluster. - Native hashing of NullType and ANSI interval values fails with "Unsupported data type in hasher" ([#7063](https://github.com/apache/datafusion-comet/issues/7063)) - Labeled high by the author for a failed native hash of NullType and ANSI interval values. It is a visible task failure rather than wrong results, so medium is defensible if the reviewer reads step 2 as native panics only. - Native element_at and map_from_arrays skip a shared subexpression on NULL rows under ANSI ([#6970](https://github.com/apache/datafusion-comet/issues/6970)) - Medium with #6971. Both make Comet return rows where Spark raises under ANSI. If the shared subexpression is common in real ANSI workloads, `priority:high` or critical may fit. ## Skipped (needs more info) None of these has a type decision to make that the guide can settle. `requires-triage` was left in place on all seven, so they reappear in the next pass until they are closed or classified. - Arrow struct writer should respect projected schema width ([#6576](https://github.com/apache/datafusion-comet/issues/6576)) - There is still no reproduction on a supported path. It has carried `bug` since the author filed it, which this pass left alone. The question from the #6578 review on 2026-10-07, which producer hands Comet a wider struct than the plan's attribute, is unanswered. - TPC-DS q54/q68: local SF100 results and distributed reproduction limits ([#6946](https://github.com/apache/datafusion-comet/issues/6946)) - A reproduction report for #6418 that says it is not a claim the issue is resolved. It is neither a bug nor a feature request. - Audit the PRs in 1.1.0 for regressions since 1.0.0 ([#6399](https://github.com/apache/datafusion-comet/issues/6399)) - An audit task, not a bug report or a feature request, and the same call as the 2026-10-12 pass. Every phase is done and the results are in #6402, so it can be closed. - Bug triage results: 2026-10-12 ([#6892](https://github.com/apache/datafusion-comet/issues/6892)) - The summary of the 2026-10-12 pass. It can be closed once reviewed. - Bug triage results: 2026-10-05 ([#6665](https://github.com/apache/datafusion-comet/issues/6665)) - The summary of the 2026-10-05 pass. It can be closed. - Bug triage results: 2026-09-28 ([#6321](https://github.com/apache/datafusion-comet/issues/6321)) - The summary of the 2026-09-28 pass. It can be closed. - Bug triage results: 2026-08-24 ([#5454](https://github.com/apache/datafusion-comet/issues/5454)) - The summary of the 2026-08-24 pass. It can be closed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
