andygrove opened a new issue, #7707: URL: https://github.com/apache/datafusion-comet/issues/7707
Bug triage pass for 2026-11-20. - Issues processed: 17 (7 bugs, 10 enhancements), plus 1 skipped. Last week's report #7640 was left alone. - Bugs by priority this pass: critical 1, high 1, medium 2, low 3. Criteria: [Bug Triage Guide](https://github.com/apache/datafusion-comet/blob/main/docs/source/contributor-guide/bug_triage.md). Labels have already been applied and `requires-triage` removed. Please spot-check the calls and close this issue when satisfied. ## Open critical bugs 26 open, 1 new this week. | Area | Open | New | | --- | ---: | ---: | | `area:Iceberg` | 8 | 0 | | `area:expressions` | 7 | 1 | | `area:scan` | 6 | 0 | | `area:writer` | 1 | 0 | | `area:shuffle` | 1 | 0 | | `area:joins` | 1 | 0 | | `area:aggregation` | 1 | 0 | | No area label | 1 | 0 | | **Total** | 26 | 1 | ### area:Iceberg - Native Iceberg scan returns NULL for nested fields of migrated tables that use a name mapping ([#6790](https://github.com/apache/datafusion-comet/issues/6790)) - Native Iceberg scan returns nested values that Spark reads as NULL in files without field ids or a name mapping ([#6794](https://github.com/apache/datafusion-comet/issues/6794)) - [EPIC] Nested types in the native Parquet and Iceberg scans: correctness and performance follow-ups ([#7040](https://github.com/apache/datafusion-comet/issues/7040)) - Native Iceberg scan applies equality deletes to the wrong column when column names differ only in case ([#7129](https://github.com/apache/datafusion-comet/issues/7129)) - Native Iceberg scan drops -0.0 rows for floating-point predicates ([#7136](https://github.com/apache/datafusion-comet/issues/7136)) - Native Iceberg scan drops NULL rows for NOT (col <=> value) ([#7137](https://github.com/apache/datafusion-comet/issues/7137)) - Native Iceberg scan drops rows from data files that store a narrower type than the column ([#7138](https://github.com/apache/datafusion-comet/issues/7138)) - Native Iceberg scan returns NULL for an INT96 column read through a name mapping when other columns are selected ([#7212](https://github.com/apache/datafusion-comet/issues/7212)) ### area:expressions - **NEW** Non-ANSI divide, div and remainder, and ln, log10 and log2 return wrong results for a nondeterministic operand ([#7671](https://github.com/apache/datafusion-comet/issues/7671)) - Comet evaluates a nondeterministic divisor or argument twice, so `/`, `div`, `%`, `ln`, `log10` and `log2` return wrong values with ANSI off. - Date-to-timestamp casts can overflow or panic for wide dates ([#5456](https://github.com/apache/datafusion-comet/issues/5456)) - size, arrays_zip, map_from_arrays and array_append return wrong answers for a nondeterministic child ([#5781](https://github.com/apache/datafusion-comet/issues/5781)) - [EPIC] Match Spark's -0.0 and NaN semantics systematically instead of per expression ([#6385](https://github.com/apache/datafusion-comet/issues/6385)) - Native map construction doesn't match Spark 4.0+ float key normalization (-0.0 keys, missing DUPLICATED_MAP_KEY) ([#6549](https://github.com/apache/datafusion-comet/issues/6549)) - With ANSI mode on, Comet's filter and projection return rows where Spark raises `DIVIDE_BY_ZERO` ([#7480](https://github.com/apache/datafusion-comet/issues/7480)) - Native string to decimal cast fails on a string with more than 38 digits: NULL or `CAST_INVALID_INPUT` where Spark returns a value ([#7581](https://github.com/apache/datafusion-comet/issues/7581)) ### area:scan - Checked TIMESTAMP_MILLIS overflow for nested fields and nested-predicate scans is blocked on DataFusion nested-field pruning ([#5553](https://github.com/apache/datafusion-comet/issues/5553)) - Track nested-field statistics pruning and TIMESTAMP_MILLIS overflow parity in filtered scans ([#5739](https://github.com/apache/datafusion-comet/issues/5739)) - [EPIC] In-memory cache audit: correctness and performance follow-ups for InMemoryTableScanExec support ([#6834](https://github.com/apache/datafusion-comet/issues/6834)) - Native Parquet scan returns wrong INT96 timestamps inside legacy-format lists ([#7029](https://github.com/apache/datafusion-comet/issues/7029)) - Native Parquet scan returns NULL for nested BINARY values that are not valid UTF-8 when they are read as STRING ([#7032](https://github.com/apache/datafusion-comet/issues/7032)) - Native object-store errors and debug logs print whole URLs outside the Azure store ([#7128](https://github.com/apache/datafusion-comet/issues/7128)) ### area:writer - Iceberg `truncate` on integers with widths above 2^30: native kernel and native writer compute different partition values ([#7442](https://github.com/apache/datafusion-comet/issues/7442)) ### area:shuffle - [EPIC] Consistent handling of invalid UTF-8 in native StringType (ingress policy) ([#4764](https://github.com/apache/datafusion-comet/issues/4764)) ### area:joins - forceShuffledHashJoin leaves sortWithinPartitions output unsorted and fails partitioned inserts ([#6770](https://github.com/apache/datafusion-comet/issues/6770)) ### area:aggregation - Match Spark ObjectHashAggregate decimal AVG buffer semantics ([#5509](https://github.com/apache/datafusion-comet/issues/5509)) ### No area label - Native broadcast nested loop joins report the streamed side's order without keeping it ([#6898](https://github.com/apache/datafusion-comet/issues/6898)) ## Needs a decision ### Escalations to consider - ci: fork runs never get a Maven cache, so Maven Central throttles them (429) ([#7702](https://github.com/apache/datafusion-comet/issues/7702)) - Fork runs fail at random across at least six contributors; the guide says a `priority:low` CI flake that blocks PR merges consistently goes to `priority:medium`. ### Skipped - needs more info - TPC-DS q54/q68: local SF100 results and distributed reproduction limits ([#6946](https://github.com/apache/datafusion-comet/issues/6946)) - A benchmark reproduction report with a tooling finding fixed by a draft PR; it is not clear whether this is a Comet defect or a research note. ## Other new bugs ### priority:high - Native shuffle fails with ClassCastException when its child is Spark's RowToColumnarExec ([#7703](https://github.com/apache/datafusion-comet/issues/7703)) - A `ClassCastException` aborts the stage when native shuffle sits on a Spark `RowToColumnarExec`, on a supported path (seen on EMR). <details> <summary>priority:medium (2)</summary> - ANSI: native `IN` evaluates candidates that Spark skips when the value is NULL or can fail ([#7675](https://github.com/apache/datafusion-comet/issues/7675)) - Offset overflow - Operators that expand a large string scalar without a batch-size bound ([#7689](https://github.com/apache/datafusion-comet/issues/7689)) </details> <details> <summary>priority:low (3)</summary> - Native regexp_extract and regexp_extract_all error on an out-of-range idx even when no row matches ([#7685](https://github.com/apache/datafusion-comet/issues/7685)) - SQL file tests with a tolerance pass when only one side is NaN ([#7672](https://github.com/apache/datafusion-comet/issues/7672)) - ci: fork runs never get a Maven cache, so Maven Central throttles them (429) ([#7702](https://github.com/apache/datafusion-comet/issues/7702)) </details> ## Enhancements <details> <summary>10 enhancements</summary> - Support CheckOverflowInTableInsert natively ([#7656](https://github.com/apache/datafusion-comet/issues/7656)) - Skip `CheckOverflow` around decimal `+`, `-`, and `*` whose result type always fits ([#7659](https://github.com/apache/datafusion-comet/issues/7659)) - Grouped decimal SUM and AVG check the running sum's precision on every input row ([#7660](https://github.com/apache/datafusion-comet/issues/7660)) - Native Iceberg writes: support S3FileIO's AssumeRoleAwsClientFactory (client.factory + client.assume-role.region) ([#7667](https://github.com/apache/datafusion-comet/issues/7667)) - Pass `fail_on_error` to datafusion-spark functions through `ConfigOptions` ([#7669](https://github.com/apache/datafusion-comet/issues/7669)) - docs: serve the newest release at user-guide/latest and the main snapshot at user-guide/dev ([#7684](https://github.com/apache/datafusion-comet/issues/7684)) - Charge native Parquet scan memory to the task memory pool ([#7688](https://github.com/apache/datafusion-comet/issues/7688)) - perf: divide once per row in decimal `/` and `div` ([#7694](https://github.com/apache/datafusion-comet/issues/7694)) - perf: compute try-mode integer `+`, `-`, and `*` without a per-row overflow branch ([#7695](https://github.com/apache/datafusion-comet/issues/7695)) - perf: divide by precomputed divisors from arrow-rs ([#7697](https://github.com/apache/datafusion-comet/issues/7697)) </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
