JeonDaehong opened a new issue, #3464: URL: https://github.com/apache/iceberg-rust/issues/3464
### Apache Iceberg Rust version main (ba849a6ed) ### Describe the bug arrow-rs truncates Parquet footer statistics to 64 bytes by default (`DEFAULT_STATISTICS_TRUNCATE_LENGTH = Some(64)`), and `MinMaxColAggregator::update` (`writer/file_writer/parquet_writer.rs:374`) skips any statistic that isn't exact. So a string or binary column whose min or max is longer than 64 bytes gets no `lower_bounds` / `upper_bounds` entry at all. parquet-java does not truncate footer statistics by default (`ParquetProperties.DEFAULT_STATISTICS_TRUNCATE_LENGTH = Integer.MAX_VALUE`), so iceberg-java keeps these bounds. The most visible case is `PositionDeleteFileWriter`: data file paths are usually longer than 64 bytes, so position delete files lose their `file_path` bounds and can't be recognized as referencing a single data file. Java writes those two columns in `Full` mode for exactly this (`MetricsConfig.forPositionDelete`). The delete file index then applies such a delete file to every data file in its partition instead of the one it references. ### To Reproduce `PositionDeleteFileWriter` over `ParquetWriterBuilder::new(WriterProperties::builder().build(), position_delete_schema())`, writing two deletes for one data file path: | path length | `file_path` lower/upper bounds | |---|---| | 40 bytes | present, equal | | 70 bytes | missing | | 200 bytes | missing | With `.set_statistics_truncate_length(None)`, all three have equal bounds. ### Expected behavior The bounds are kept, as in Java. One option is to turn off arrow-rs statistics truncation in `ParquetWriter`, since iceberg-rust stores bounds itself and truncation by metrics mode belongs there (#2362); at least position delete files need untruncated `file_path` bounds. ### Willingness to contribute I can contribute a fix for this bug independently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
