andygrove commented on PR #5677: URL: https://github.com/apache/datafusion-comet/pull/5677#issuecomment-5622520035
## CI failure summary: `d2383b1` ([run 34491795459](https://github.com/apache/datafusion-comet/actions/runs/34491795459)) **18 failing Iceberg tests**, down from 86 at the `d86d3b5` baseline ([run 33995765073](https://github.com/apache/datafusion-comet/actions/runs/33995765073)). They land in 10 of the 24 Iceberg shard jobs. Separately, the 5 `PR Build [scans]` jobs each fail the same 11 `CometIcebergWriteActionSuite` plan assertions. | Count | Cause | Suites | Issue | |---:|---|---|---| | 8 | Partition path for a `double` column is rendered as a full decimal expansion (`d=17976931348623157000...000`, 309 characters), so the directory name exceeds `NAME_MAX` and the write dies with `Failed to finish parquet writer ... invalid filename, source: File name too long (os error 36)` | `TestSparkDataFile.testValueConversionWithEmptyStats` and `.testValueConversionPartitionedTable`, on all four Iceberg versions | not filed | | 7 | Metadata table returns rows in a different order (`row 1 contents should match, expected: 1 but was: 3`) | `TestMetadataTablesWithPartitionEvolution.testPartitionColumnNamedPartition` | #5776, PR #5810 open | | 3 | Compaction after partition evolution yields `[2, 2, 0]` where Iceberg expects `[1, 2, 1]`: the 2 data files are not rewritten into 1 and the dangled position delete file is not removed | `TestRewriteDataFilesAction.testRemoveDangledPositionDeletesPartitionEvolution`, format version 2 only (version 3 passes) | not filed | | 11 (x5 jobs) | Plan assertions expect the JVM two-operator `IcebergWriteExec`, the plan now has `CometIcebergWrite` | `CometIcebergWriteActionSuite` in every `[scans]` job | test staleness, belongs on the #5644 checklist | Counts are test-case instances summed over jobs, so a single root cause is counted once per Iceberg version and once per parameterization. Per Iceberg version: 1.8/3.4 has 2, 1.9/3.5 has 5, 1.10/3.5 has 5, 1.11/4.1 has 6. ### What cleared since the baseline The branch merged main twice and picked up five fixes, which between them account for all 68 failures that went away: - #5689 (PR #5696), the dropped `ColumnarToRow` under the native write. **All four `iceberg-spark-extensions` jobs are green now**, where the baseline had 22 failures across them. This was the AQE-off failure whose count swung run to run, so the swing is gone with it. - #5690 (PR #5780), file rolling on iceberg-java's 1000-row grid. `TestSparkDataWrite` passes in full, 135 cases green, 0 failing, against 24 failing at baseline. - #5698 (PR #5779), the exception type for unclustered input to a clustered writer. `TestRequiredDistributionAndOrdering` is 64 green, 0 failing, against 32 failing at baseline. - #5691 and #5693 and #5694 (all PR #5729), the partition-spec-evolution panics. `TestAlterTablePartitionFields` passes. - #5758 (PR #5759), unknown partition transforms on the scan side. `TestForwardCompatibility` passes. ### The two unfiled causes **Double partition values.** iceberg-java writes `d=1.7976931348623157E308` because `Double.toString` switches to scientific notation outside `1e-3 .. 1e7`; the native path writes all 309 digits, and Linux rejects a path component over 255 bytes. The other failing case is `Double.MIN_VALUE`, written as `0.000...005` with 324 digits. This is a partition-path formatting divergence in the native writer, not a filesystem limitation we can configure around, and it will hit any table partitioned by an identity `double` or `float` whose values fall outside that range. These 8 cases were previously masked: at baseline the same 8 failed on the #5694 `Option::unwrap` panic, which fired earlier in the same code path. **Dangled position deletes after partition evolution.** Baseline grouped this with #5690, and the other `TestRewriteDataFilesAction` case in that group (`testBinPackCombineMediumFiles`) is now green, so the remaining assertion is a separate problem rather than residue of the file-size work. The rewrite sees 2 data files where it should see 1 and leaves the dangled delete in place, only at format version 2. ### Reading the numbers The `[scans]` failures are a test-expectation problem, not a writer bug: those 11 cases assert the JVM writer appears in the plan and the native writer has taken over. The remaining Iceberg failures now cluster into three narrow, specific defects rather than the broad breakage the first run showed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
