andygrove opened a new issue, #7574: URL: https://github.com/apache/datafusion-comet/issues/7574
Bug triage pass for 2026-11-09. - Issues processed: 78 (34 bugs, 44 enhancements) - Bugs by priority: critical 5, high 7, medium 16, low 6 - Skipped: 3 Criteria: [Bug Triage Guide](https://github.com/apache/datafusion-comet/blob/main/docs/source/contributor-guide/bug_triage.md). Labels have already been applied and `requires-triage` removed. Please spot-check the calls and close this issue when satisfied. Many issues arrived already carrying a type (and, for bugs, a priority) label; those were kept as they were. ## Bugs ### priority:critical - Iceberg `truncate` on integers with widths above 2^30: native kernel and native writer compute different partition values ([#7442](https://github.com/apache/datafusion-comet/issues/7442)) - Area labels: `area:expressions`, `area:writer` - Rationale: Native kernel and native writer compute different Iceberg partition values for integer truncate widths above 2^30, a silent mismatch (rare widths; see escalations). - read_side_padding ignores the pad string for dictionary inputs in lpad and rpad ([#7453](https://github.com/apache/datafusion-comet/issues/7453)) - Area labels: `area:expressions` - Rationale: `lpad`/`rpad` on dictionary input pads with spaces instead of the pad string, a silent wrong result. - With ANSI mode on, Comet's filter and projection return rows where Spark raises `DIVIDE_BY_ZERO` ([#7480](https://github.com/apache/datafusion-comet/issues/7480)) - Area labels: `area:expressions` - Rationale: Under ANSI, Comet returns rows where Spark raises `DIVIDE_BY_ZERO`, so it can silently differ from Spark (see escalations). - Offset overflow - Native columnar-to-row conversion wraps row offsets past 2 GiB ([#7494](https://github.com/apache/datafusion-comet/issues/7494)) - Area labels: `area:ffi` - Rationale: Already labeled `bug` with this priority; kept as is. - Spark window over a doubted input order gets a sort by its own keys only, and operators above it return wrong rows ([#7553](https://github.com/apache/datafusion-comet/issues/7553)) - Area labels: none - Rationale: Operators above a Spark window return wrong rows silently, which is `priority:critical` step 1 of the decision tree. ### priority:high - Native Parquet scan reads invalid UTF-8 in nested BINARY values as unvalidated strings on DataFusion 56 ([#7469](https://github.com/apache/datafusion-comet/issues/7469)) - Area labels: `area:scan` - Rationale: Already labeled `bug` with this priority; kept as is. - Native timestamp casts fail or panic for years beyond chrono's range ([#7471](https://github.com/apache/datafusion-comet/issues/7471)) - Area labels: `area:expressions` - Rationale: Native timestamp casts panic or raise for years beyond chrono's range, which is a native panic. - Offset overflow - Shuffle read panics expanding dictionary blocks from the JVM columnar shuffle ([#7496](https://github.com/apache/datafusion-comet/issues/7496)) - Area labels: `area:shuffle` - Rationale: Already labeled `bug` with this priority; kept as is. - Offset overflow - JVM columnar shuffle cuts row-to-Arrow batches by row count only ([#7497](https://github.com/apache/datafusion-comet/issues/7497)) - Area labels: `area:shuffle` - Rationale: Already labeled `bug` with this priority; kept as is. - Offset overflow - String kernels can produce more than 2 GiB per batch, and some reserve it up front ([#7500](https://github.com/apache/datafusion-comet/issues/7500)) - Area labels: `area:expressions` - Rationale: Already labeled `bug` with this priority; kept as is. - Offset overflow - Comet's grouped accumulators build one array for all emitted groups ([#7501](https://github.com/apache/datafusion-comet/issues/7501)) - Area labels: `area:aggregation` - Rationale: Already labeled `bug` with this priority; kept as is. - `GetArrayItem` fails on a non-INT index: `could not cast array of type Int64 ...` ([#7551](https://github.com/apache/datafusion-comet/issues/7551)) - Area labels: `area:expressions` - Rationale: A valid query with a non-INT array index fails with a native exception on a user's real workload, which is major functional breakage on a supported path. ### priority:medium - Native Iceberg scan with AQE DPP fails when the DPP build side is a storage-partitioned join ([#7441](https://github.com/apache/datafusion-comet/issues/7441)) - Area labels: `area:scan` - Rationale: Assertion failure for one narrow plan shape (DPP over a storage-partitioned join) with the opt-in native Iceberg scan; the Spark scan is a workaround. - Native lpad and rpad fail when the length argument is a NULL scalar, such as a scalar subquery that returns no rows ([#7452](https://github.com/apache/datafusion-comet/issues/7452)) - Area labels: `area:expressions` - Rationale: `lpad`/`rpad` fail on a NULL scalar length such as an empty scalar subquery; rare input with a fallback workaround. - ANSI: comparisons evaluate the right operand on rows where the left one is NULL ([#7468](https://github.com/apache/datafusion-comet/issues/7468)) - Area labels: `area:expressions` - Rationale: Under ANSI, comparisons evaluate the right operand where Spark skips it and raise false errors; visible failure. - Native sort evaluates later sort keys on rows Spark's ordering skips, raising false ANSI errors ([#7470](https://github.com/apache/datafusion-comet/issues/7470)) - Area labels: none - Rationale: Under ANSI, native sort evaluates later keys on rows Spark skips and raises false errors; visible failure. - `spark.comet.expression.<Name>.enabled=false` has no effect on `parse_url`, `to_json` and `json_array_length` on Spark 4.1 ([#7478](https://github.com/apache/datafusion-comet/issues/7478)) - Area labels: `area:expressions` - Rationale: `spark.comet.expression.<Name>.enabled=false` has no effect for three expressions on Spark 4.1; results are correct and a workaround is documented in the issue. - `rand` and `randn` with an integer seed fall back to Spark without a fallback reason ([#7479](https://github.com/apache/datafusion-comet/issues/7479)) - Area labels: `area:expressions` - Rationale: `rand`/`randn` with an integer seed falls back to Spark with no reason recorded; a missing-support gap with a workaround. - CometShuffleManager treats Spark's own shuffles differently from SortShuffleManager ([#7488](https://github.com/apache/datafusion-comet/issues/7488)) - Area labels: `area:shuffle` - Rationale: `CometShuffleManager` adds a row copy and leaks per-shuffle state on the executor heap; no wrong results or crash. - Native Iceberg scan copies the whole scan's delete-file metadata into every task, causing executor OOM on large merge-on-read tables ([#7490](https://github.com/apache/datafusion-comet/issues/7490)) - Area labels: `area:scan` - Rationale: Opt-in native Iceberg scan hits executor OOM on very large merge-on-read tables; a broken feature with the Spark scan as workaround. - Offset overflow - Explode copies columns past 2 GiB and never splits one long row ([#7495](https://github.com/apache/datafusion-comet/issues/7495)) - Area labels: `area:expressions` - Rationale: Already labeled `bug` with this priority; kept as is. - Offset overflow - Codegen dispatcher writes each batch's output into one Arrow vector ([#7498](https://github.com/apache/datafusion-comet/issues/7498)) - Area labels: `area:expressions`, `area:udf` - Rationale: Already labeled `bug` with this priority; kept as is. - Offset overflow - Native Parquet scan expands flat dictionary columns and reads twice Spark's batch rows ([#7499](https://github.com/apache/datafusion-comet/issues/7499)) - Area labels: `area:scan` - Rationale: Already labeled `bug` with this priority; kept as is. - Offset overflow - Spark-to-Arrow conversion cuts batches by row count only ([#7502](https://github.com/apache/datafusion-comet/issues/7502)) - Area labels: `area:scan` - Rationale: Already labeled `bug` with this priority; kept as is. - Native TopK evaluates sort keys for a single-row input, raising false ANSI errors ([#7527](https://github.com/apache/datafusion-comet/issues/7527)) - Area labels: none - Rationale: Under ANSI, a single-row TopK raises a false error that Spark does not; visible failure, limited to ANSI mode. - ANY_VALUE memory regression in Comet 1.2.0 ([#7541](https://github.com/apache/datafusion-comet/issues/7541)) - Area labels: `area:aggregation` - Rationale: Comet 1.2.0 fails `ANY_VALUE(array<string>)` with an oversized memory reservation; disabling native aggregates is a workaround (see escalations). - Backport ANSI skipped-operand fix for arithmetic and concat to branch-1.2 ([#7542](https://github.com/apache/datafusion-comet/issues/7542)) - Area labels: `area:expressions` - Rationale: `branch-1.2` raises false `CAST_INVALID_INPUT` errors for queries Spark answers; the failure is visible and Spark fallback is a workaround. - [Bug] hll_sketch_agg, hll_union and hll_union_agg write compact sketch bytes where Spark writes updatable ones ([#7548](https://github.com/apache/datafusion-comet/issues/7548)) - Area labels: `area:aggregation`, `area:expressions` - Rationale: Sketch bytes differ from Spark, but the expressions are gated behind `allowIncompatible`, so users opt in knowingly; a broken feature with a workaround. ### priority:low - test: no test on main reaches the folded UnsafeArrayData literal fix from #6666 ([#7458](https://github.com/apache/datafusion-comet/issues/7458)) - Area labels: `area:expressions` - Rationale: No test reaches the #6666 fix, so a regression would go unnoticed; test-only. - `CometBenchmarkBase.runExpressionBenchmark` checks the initial AQE plan, not the executed plan ([#7481](https://github.com/apache/datafusion-comet/issues/7481)) - Area labels: none - Rationale: Benchmark tooling checks the wrong plan; test/tooling only. - Offset overflow - Lower-risk sites (opt-in features, unrealistic sizes, upstream panics) ([#7503](https://github.com/apache/datafusion-comet/issues/7503)) - Area labels: none - Rationale: Already labeled `bug` with this priority; kept as is. - [Bug] hll_* errors surface as Comet errors instead of HLL_INVALID_INPUT_SKETCH_BUFFER / HLL_UNION_DIFFERENT_LG_K ([#7549](https://github.com/apache/datafusion-comet/issues/7549)) - Area labels: `area:expressions` - Rationale: Both engines fail on the same inputs; only the error class and SQLSTATE differ, which is cosmetic. - ParquetWriterExec's derived Debug prints object store options, which can hold secret keys ([#7558](https://github.com/apache/datafusion-comet/issues/7558)) - Area labels: `area:writer` - Rationale: Derived `Debug` would print object store secrets, but Comet does not format the operator with `{:?}` today, so nothing leaks yet; low (see escalations). - Native Iceberg errors print data and delete file paths with their credentials ([#7559](https://github.com/apache/datafusion-comet/issues/7559)) - Area labels: none - Rationale: Error messages can print credentials embedded in Iceberg file paths, but locations rarely carry credentials and nothing is wrong with results, so low (see escalations). ## Enhancements - Comet Native Executors: let a fragment's partial aggregate stop aggregating, including AVG ([#7406](https://github.com/apache/datafusion-comet/issues/7406)) - Area labels: `area:aggregation` - Rationale: Already labeled `enhancement`; kept as is. - Comet Native Executors: choose partition counts for pipelined mode ([#7407](https://github.com/apache/datafusion-comet/issues/7407)) - Area labels: none - Rationale: Already labeled `enhancement`; kept as is. - Comet Native Executors: broadcast a small build side in pipelined mode ([#7408](https://github.com/apache/datafusion-comet/issues/7408)) - Area labels: none - Rationale: Already labeled `enhancement`; kept as is. - Comet Native Executors: TPC-H q17 in pipelined mode takes 277 s where Ballista takes 154 s ([#7409](https://github.com/apache/datafusion-comet/issues/7409)) - Area labels: none - Rationale: Already labeled `enhancement`; kept as is. - Create 1.3.0 Release (Dec 2026) ([#7410](https://github.com/apache/datafusion-comet/issues/7410)) - Area labels: none - Rationale: Already labeled `enhancement`; kept as is. - Enable the Comet in-memory cache (`InMemoryTableScan`) by default ([#7425](https://github.com/apache/datafusion-comet/issues/7425)) - Area labels: none - Rationale: Already labeled `enhancement`; kept as is. - Enable `LocalTableScan` conversion by default ([#7426](https://github.com/apache/datafusion-comet/issues/7426)) - Area labels: none - Rationale: Already labeled `enhancement`; kept as is. - Support Iceberg's `MergeRowsExec` on Spark 3.4 and 3.5 ([#7427](https://github.com/apache/datafusion-comet/issues/7427)) - Area labels: none - Rationale: Already labeled `enhancement`; kept as is. - Native Iceberg writes for CTAS / RTAS on Spark 3.4 ([#7428](https://github.com/apache/datafusion-comet/issues/7428)) - Area labels: `area:writer` - Rationale: Already labeled `enhancement`; kept as is. - Native JSON file scan ([#7429](https://github.com/apache/datafusion-comet/issues/7429)) - Area labels: `area:scan` - Rationale: Already labeled `enhancement`; kept as is. - Native CSV scan for V1 `FileSourceScanExec` ([#7430](https://github.com/apache/datafusion-comet/issues/7430)) - Area labels: `area:scan` - Rationale: Already labeled `enhancement`; kept as is. - Native scans for Iceberg metadata tables ([#7431](https://github.com/apache/datafusion-comet/issues/7431)) - Area labels: `area:scan` - Rationale: Already labeled `enhancement`; kept as is. - Support `from_protobuf` (`ProtobufDataToCatalyst`) ([#7432](https://github.com/apache/datafusion-comet/issues/7432)) - Area labels: `area:expressions` - Rationale: Already labeled `enhancement`; kept as is. - Support `input_file_name()` and `input_file_block_*` with native scans ([#7433](https://github.com/apache/datafusion-comet/issues/7433)) - Area labels: `area:scan` - Rationale: Already labeled `enhancement`; kept as is. - Window `min` / `max` over string and binary input ([#7434](https://github.com/apache/datafusion-comet/issues/7434)) - Area labels: `area:aggregation` - Rationale: Already labeled `enhancement`; kept as is. - `percentile` with an array of percentages or a frequency argument ([#7435](https://github.com/apache/datafusion-comet/issues/7435)) - Area labels: `area:aggregation` - Rationale: Already labeled `enhancement`; kept as is. - Native text file writes ([#7436](https://github.com/apache/datafusion-comet/issues/7436)) - Area labels: `area:writer` - Rationale: Already labeled `enhancement`; kept as is. - Native JSON file writes ([#7437](https://github.com/apache/datafusion-comet/issues/7437)) - Area labels: `area:writer` - Rationale: Already labeled `enhancement`; kept as is. - Native CSV file writes ([#7438](https://github.com/apache/datafusion-comet/issues/7438)) - Area labels: `area:writer` - Rationale: Already labeled `enhancement`; kept as is. - `approx_percentile` / `percentile_approx` on non-numeric input ([#7439](https://github.com/apache/datafusion-comet/issues/7439)) - Area labels: `area:aggregation` - Rationale: Already labeled `enhancement`; kept as is. - `max_by` / `min_by` with variable-length value or ordering types ([#7440](https://github.com/apache/datafusion-comet/issues/7440)) - Area labels: `area:aggregation` - Rationale: Already labeled `enhancement`; kept as is. - Comet's TPC-DS speedup is smaller on Spark 4.2.0 than on Spark 4.1.3; investigate and improve Spark 4.2 acceleration ([#7451](https://github.com/apache/datafusion-comet/issues/7451)) - Area labels: none - Rationale: Investigation to improve Spark 4.2 acceleration; Comet results are correct and the gap is mostly a faster Spark baseline. - Pair the spark-4.2 profile with Delta 4.4 and run the Delta contrib tests on it ([#7455](https://github.com/apache/datafusion-comet/issues/7455)) - Area labels: `area:ci` - Rationale: Request to pair the `spark-4.2` profile with Delta 4.4 and run the Delta contrib tests on it. - perf: native Parquet scans make two S3 requests for a small file and fetch large merged ranges with one GET ([#7459](https://github.com/apache/datafusion-comet/issues/7459)) - Area labels: `area:scan` - Rationale: Already labeled `enhancement`; kept as is. - Native Parquet write falls back on int96RebaseModeInWrite=LEGACY even when timestamps are not written as INT96 ([#7463](https://github.com/apache/datafusion-comet/issues/7463)) - Area labels: `area:writer` - Rationale: Already labeled `enhancement`; kept as is. - Comet Native Executors: report each stage's Comet block metrics to Ballista ([#7465](https://github.com/apache/datafusion-comet/issues/7465)) - Area labels: none - Rationale: Already labeled `enhancement`; kept as is. - Reduce per-partition overhead for ROW_NUMBER windows ([#7466](https://github.com/apache/datafusion-comet/issues/7466)) - Area labels: none - Rationale: Already labeled `enhancement`; kept as is. - perf: file open and footer reads dominate native Parquet scan time on TPC-DS q54 ([#7467](https://github.com/apache/datafusion-comet/issues/7467)) - Area labels: `area:scan` - Rationale: Performance work on native Parquet scan open time; nothing is incorrect. - perf: tasks that share a decoded broadcast each reserve the whole build side ([#7472](https://github.com/apache/datafusion-comet/issues/7472)) - Area labels: none - Rationale: Already labeled `enhancement`; kept as is. - Native hash shuffle gathers rows one at a time even when a whole batch goes to one partition ([#7474](https://github.com/apache/datafusion-comet/issues/7474)) - Area labels: `area:shuffle` - Rationale: Already labeled `enhancement`; kept as is. - Native CartesianProductExec: multi-partition inputs and spilling ([#7476](https://github.com/apache/datafusion-comet/issues/7476)) - Area labels: none - Rationale: Already labeled `enhancement`; kept as is. - perf: broadcasts with dictionary-encoded columns are neither coalesced nor shared ([#7486](https://github.com/apache/datafusion-comet/issues/7486)) - Area labels: none - Rationale: Already labeled `enhancement`; kept as is. - perf: probe tasks wait for the shared broadcast decode, and every wave of tasks decodes again ([#7487](https://github.com/apache/datafusion-comet/issues/7487)) - Area labels: none - Rationale: Already labeled `enhancement`; kept as is. - Support Iceberg 1.12.0 and run the Iceberg suites on Spark 4.2 ([#7493](https://github.com/apache/datafusion-comet/issues/7493)) - Area labels: `area:Iceberg`, `area:ci` - Rationale: Already labeled `enhancement`; kept as is. - Research: relate Spark 4.3 Arrow cache serializer to Comet in-memory cache ([#7517](https://github.com/apache/datafusion-comet/issues/7517)) - Area labels: none - Rationale: Already labeled `enhancement`; kept as is. - [EPIC] Support Spark 4.3 ([#7519](https://github.com/apache/datafusion-comet/issues/7519)) - Area labels: none - Rationale: Already labeled `enhancement`; kept as is. - Native Iceberg scan builds every partition's plan data on the driver up front ([#7523](https://github.com/apache/datafusion-comet/issues/7523)) - Area labels: `area:scan` - Rationale: Already labeled `enhancement`; kept as is. - Support a literal str argument in native lpad/rpad ([#7525](https://github.com/apache/datafusion-comet/issues/7525)) - Area labels: `area:expressions` - Rationale: Request to take the native path for a literal `str` argument instead of codegen dispatch; results are already correct. - Native lambdas: count a widening cast as unable to raise under ANSI mode ([#7535](https://github.com/apache/datafusion-comet/issues/7535)) - Area labels: `area:expressions` - Rationale: Already labeled `enhancement`; kept as is. - Comet Native Executors: run a query in Spark when Comet is turned off ([#7557](https://github.com/apache/datafusion-comet/issues/7557)) - Area labels: none - Rationale: Already labeled `enhancement`; kept as is. - Support Delta FileSizeHistogramAgg ([#7570](https://github.com/apache/datafusion-comet/issues/7570)) - Area labels: `area:aggregation` - Rationale: Request to support a new Delta aggregate natively. - Support Delta DeletedRecordCountsHistogramAgg ([#7571](https://github.com/apache/datafusion-comet/issues/7571)) - Area labels: `area:aggregation` - Rationale: Request to support a new Delta aggregate natively. - Support Delta BitmapAggregator with Portable encoding ([#7572](https://github.com/apache/datafusion-comet/issues/7572)) - Area labels: `area:aggregation` - Rationale: Request to support a new Delta aggregate natively. - Support Delta InterleaveBits for one or two INT inputs ([#7573](https://github.com/apache/datafusion-comet/issues/7573)) - Area labels: `area:expressions` - Rationale: Request to support a new Delta expression natively. ## Escalations to consider - Native Iceberg errors print data and delete file paths with their credentials ([#7559](https://github.com/apache/datafusion-comet/issues/7559)) - The guide lists security vulnerabilities as `priority:critical`. Labeled `priority:low` because the reporter calls the risk low and the path has to carry credentials, but a reviewer may want to raise it. [#7558](https://github.com/apache/datafusion-comet/issues/7558) is the same class (no leak today). - ANY_VALUE memory regression in Comet 1.2.0 ([#7541](https://github.com/apache/datafusion-comet/issues/7541)) - A regression from 1.1.0 that fails the query; `priority:medium` has a workaround, but escalate to `priority:high` if other users report it. - With ANSI mode on, Comet's filter and projection return rows where Spark raises `DIVIDE_BY_ZERO` ([#7480](https://github.com/apache/datafusion-comet/issues/7480)) - Labeled `priority:critical` per decision-tree step 1 (output differs from Spark with no error), but it only affects queries that Spark itself fails under ANSI; a reviewer may lower it. - Iceberg `truncate` on integers with widths above 2^30: native kernel and native writer compute different partition values ([#7442](https://github.com/apache/datafusion-comet/issues/7442)) - Labeled `priority:critical` per step 1 (wrong partition values), but widths above 2^30 are unusual; a reviewer may lower it. - Spark window over a doubted input order gets a sort by its own keys only, and operators above it return wrong rows ([#7553](https://github.com/apache/datafusion-comet/issues/7553)) - Silent wrong rows on main; no area label in the guide fits. ## Skipped — needs more info - Bug triage results: 2026-11-02 ([#7403](https://github.com/apache/datafusion-comet/issues/7403)) - A previous triage summary issue, not a bug or enhancement. `requires-triage` was left in place; a reviewer can close it. - Bug triage results: 2026-10-26 ([#7230](https://github.com/apache/datafusion-comet/issues/7230)) - A previous triage summary issue, not a bug or enhancement. `requires-triage` was left in place; a reviewer can close it. - TPC-DS q54/q68: local SF100 results and distributed reproduction limits ([#6946](https://github.com/apache/datafusion-comet/issues/6946)) - A reproduction report that states it does not claim #6418 is resolved; it is not clearly a bug or an enhancement. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
