JingsongLi commented on PR #477: URL: https://github.com/apache/paimon-rust/pull/477#issuecomment-4923680619
Thanks for the update. I think there are still a couple of Java compatibility gaps here. 1. The manifest stats mode is currently hard-coded to the default truncate(16) behavior. In Java, stats collectors are created from `metadata.stats-mode`, `fields.<name>.stats-mode`, and `metadata.stats-keep-first-n-columns`. With this implementation, a table configured with `metadata.stats-mode = none/counts/full` or a field-level override would still get truncate(16)-style stats in Rust-written manifests. Could we thread the table stats options into this extraction path and apply the same collector semantics as Java? 2. `supports_manifest_value_stats` currently skips unsupported min/max types completely. Java does not do that: for unsupported Parquet min/max types such as BINARY/VARBINARY, `ParquetSimpleStatsExtractor` still returns `SimpleColStats(null, null, nullCount)`, and dense stats storage keeps the column when the null count is known. The current Rust behavior drops those columns from `value_stats_cols`, so their null counts are lost. Also, Java supports TIME_WITHOUT_TIME_ZONE through IntStatistics, while Rust's conversion code has Time support but the support list excludes `DataType::Time`. Could we keep null counts for unsupported min/max columns and include Time to match Java? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
