Visorgood opened a new issue, #7639: URL: https://github.com/apache/datafusion-comet/issues/7639
### What is the problem the feature request solves? Part of #5644. One of the conditions for enabling native Iceberg writes by default is manifest metrics parity per data type on the pinned Iceberg versions (1.5.2 is dropped, per the discussion in #5644). Readers such as Spark and Trino prune data files using these metrics, so a wrong bound or count from the native writer can make a reader skip a file that has matching rows. `CometIcebergWriteActionSuite` already compares native and iceberg-java metrics through `manifestMetricsParity`, but only for some types and metrics: - bounds for `date`, `timestamp`, `timestamp_ntz`, `decimal(9,2)`, `decimal(38,10)`, `float` and `binary`, and value counts for `boolean` and `bigint`; - string bound truncation, metrics mode `none`, and sorted-column bounds under `counts`; - NaN counts for `float` and `double`, in separate tests. Not covered yet: null counts across types; bounds for `int`, `bigint`, `double`, `boolean` and short strings; INT64-backed decimals (precision 10 to 18); all-null columns; and leaf columns inside structs, lists and maps. ### Describe the potential solution Parity tests per group of types, built on `manifestMetricsParity`. Each one writes the same rows, including nulls and NaN where they apply, through the native and the iceberg-java writer, and compares per column `value_count`, `null_value_count`, `nan_value_count`, `lower_bound` and `upper_bound` from `readable_metrics`. - Types: `boolean`, `int`, `bigint`, `float`, `double`, decimals backed by INT32, INT64 and fixed-length binary, `date`, `timestamp`, `timestamp_ntz`, `string`, `binary`, and leaf columns inside `struct`, `list` and `map` where iceberg-java records metrics for them. - `column_sizes` is left out, because the two writers lay out files differently. - Types the native writer doesn't accept yet (for example UUID, #7198) are out of scope. The tests go in Comet's own suite, so they run under each Spark profile's pinned Iceberg version: 1.8.1 (Spark 3.5), 1.10.0 (Spark 4.0) and 1.11.0 (Spark 4.1). Each mismatch they find gets its own issue, since a fix may be on the native side, and the test for that type references it until it's fixed. ### Additional context Coverage summary is from `CometIcebergWriteActionSuite` on `main` at 983173e. ### Willingness to contribute I can contribute a fix for this bug independently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
