Visorgood opened a new issue, #7639:
URL: https://github.com/apache/datafusion-comet/issues/7639

   ### What is the problem the feature request solves?
   
   Part of #5644. One of the conditions for enabling native Iceberg writes by 
default is manifest metrics parity per data type on the pinned Iceberg versions 
(1.5.2 is dropped, per the discussion in #5644). Readers such as Spark and 
Trino prune data files using these metrics, so a wrong bound or count from the 
native writer can make a reader skip a file that has matching rows.
   
   `CometIcebergWriteActionSuite` already compares native and iceberg-java 
metrics through `manifestMetricsParity`, but only for some types and metrics:
   - bounds for `date`, `timestamp`, `timestamp_ntz`, `decimal(9,2)`, 
`decimal(38,10)`, `float` and `binary`, and value counts for `boolean` and 
`bigint`;
   - string bound truncation, metrics mode `none`, and sorted-column bounds 
under `counts`;
   - NaN counts for `float` and `double`, in separate tests.
   
   Not covered yet: null counts across types; bounds for `int`, `bigint`, 
`double`, `boolean` and short strings; INT64-backed decimals (precision 10 to 
18); all-null columns; and leaf columns inside structs, lists and maps.
   
   ### Describe the potential solution
   
   Parity tests per group of types, built on `manifestMetricsParity`. Each one 
writes the same rows, including nulls and NaN where they apply, through the 
native and the iceberg-java writer, and compares per column `value_count`, 
`null_value_count`, `nan_value_count`, `lower_bound` and `upper_bound` from 
`readable_metrics`.
   
   - Types: `boolean`, `int`, `bigint`, `float`, `double`, decimals backed by 
INT32, INT64 and fixed-length binary, `date`, `timestamp`, `timestamp_ntz`, 
`string`, `binary`, and leaf columns inside `struct`, `list` and `map` where 
iceberg-java records metrics for them.
   - `column_sizes` is left out, because the two writers lay out files 
differently.
   - Types the native writer doesn't accept yet (for example UUID, #7198) are 
out of scope.
   
   The tests go in Comet's own suite, so they run under each Spark profile's 
pinned Iceberg version: 1.8.1 (Spark 3.5), 1.10.0 (Spark 4.0) and 1.11.0 (Spark 
4.1). Each mismatch they find gets its own issue, since a fix may be on the 
native side, and the test for that type references it until it's fixed.
   
   ### Additional context
   
   Coverage summary is from `CometIcebergWriteActionSuite` on `main` at 983173e.
   
   ### Willingness to contribute
   
   I can contribute a fix for this bug independently


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to