andygrove commented on PR #3470: URL: https://github.com/apache/datafusion-comet/pull/3470#issuecomment-3891779856
## CI Failure Analysis ### Summary 17 CI jobs are failing across 6 distinct root causes. Here is the breakdown: --- ### 1. `width_bucket - with range data` — Arrow type cast error (8 jobs) **Failing test:** `width_bucket - with range data` **Error:** `CometNativeException: could not cast array of type Int64 to arrow_array::array::primitive_array::PrimitiveArray<arrow_array::types::Int32Type>` **Affected jobs:** All `[expressions]` jobs on macOS (Spark 3.5, 4.0) and Ubuntu (Spark 3.5.5, 3.5.6, 3.5 native_datafusion, 3.5 native_iceberg_compat, 4.0) **Root cause:** The `width_bucket` function returns `Int64` (LongType) but the native code expects `Int32`. Likely a DataFusion 52 change in the return type of `width_bucket`. Also causes `postgreSQL/numeric.sql` to fail in spark-sql jobs (3 additional jobs) returning an empty schema. --- ### 2. Schema pruning — Native reader crash on complex nested types (3 jobs, ~44 tests each) **Failing tests:** 44 schema pruning tests involving arrays/maps with IN clause predicates (e.g., `select a single complex field array and in clause`, `select nested field from a complex map key using map_keys`, `SPARK-34638: nested column prune on generator output`, etc.) **Error:** `SparkException: Encountered error while reading file ... .snappy.parquet` through `CometScanExec -> Native.executePlan` **Affected jobs:** `spark-sql-auto-sql_core-2` on Spark 3.4.3 and 3.5.8, `spark-sql-native_datafusion-sql_core-1` on Spark 3.5.8 **Root cause:** CometScanExec's native reader fails when reading Parquet files with pruned complex/nested schemas (arrays, maps, structs) combined with IN clause predicates. --- ### 3. SPARK-40819 — Timestamp nanos precision loss (4 jobs) **Failing test:** `SPARK-40819: parquet file with TIMESTAMP(NANOS, true) (with nanosAsLong=true)` **Error:** `1668537129123 did not equal 1668537129123534758` — nanosecond precision truncated to milliseconds **Affected jobs:** `spark-sql-auto-sql_core-1` on Spark 3.4.3, 3.5.8, 4.0.1 and `spark-sql-auto-sql_core-2` on Spark 4.0.1 **Root cause:** Comet's Parquet reader loses the nanosecond portion of timestamps when `nanosAsLong=true`. --- ### 4. `select column with default value` — Fuzz test default value bug (1 job, 2 tests) **Failing tests:** `select column with default value (native shuffle)` and `(jvm shuffle)` **Error:** Expected `[true]` but got `[null]` for a boolean column with a default value **Affected jobs:** `ubuntu-latest/Spark 3.5 native_datafusion [fuzz]` **Root cause:** Default column values are not being materialized correctly — returning `null` instead of the expected default `true`. --- ### 5. Miri build failure (1 job) **Error:** `fatal error: file ... build-script-build contains outdated or invalid JSON; try cargo clean` **Root cause:** Stale Miri build cache. The `quote v1.0.44` crate's build script artifact is corrupted/outdated. This is a CI infrastructure issue — not a code issue. --- ### 6. Spark 4.0-specific failures (2 jobs, 1 test each) - **`arrays and maps ignore shredding schema`**: Native reader doesn't support Spark 4.0's Parquet variant type shredding — `CometScanExec` fails reading shredded parquet files. - **`INSERT rows, ALTER TABLE ADD COLUMNS with DEFAULTs, then SELECT them`**: `TASK_WRITE_FAILED` / `IOException: Failed to rename` — appears to be a CI flaky test (staging file rename failure). --- ### Priority Assessment | # | Issue | Jobs | Severity | |---|---|---|---| | 1 | `width_bucket` Int64→Int32 cast | 11 | High — likely DF52 API change | | 2 | Schema pruning complex types | 3 | High — ~44 tests per job | | 3 | Timestamp nanos precision | 4 | Medium | | 4 | Default column values | 1 | Medium | | 5 | Miri stale cache | 1 | Low — CI infra | | 6 | Spark 4.0 shredding + flaky rename | 2 | Low | --- > *This analysis was generated with the assistance of AI (Claude Code). Failure logs were retrieved and analyzed programmatically — manual verification of root causes is recommended.* -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
