kalayciburak commented on PR #10337: URL: https://github.com/apache/paimon/pull/10337#issuecomment-5996428618
yes, looked at both. Iceberg keeps lower/upper bounds for non-null, non-NaN values only, counts NaN separately (nan_value_counts), and does not allow NaN as a bound. -0.0 precedes +0.0, same as IEEE 754 totalOrder. The ORC hole is iceberg#1761: if the first value is NaN, ORC reports NaN for both bounds, and the evaluator treats that as unreliable so it does not skip the file. That landed in #2069. Delta drops float/double min/max when the parquet writer may omit NaN from the footer (Arrow, parquet-cpp). parquet-mr is trusted because it records NaN as the max. an unknown writer is treated as unsafe. ORC updates min/max with primitive < and >, so NaN and signed zero never replace a finite bound. we also don't have a nan count on these file stats. dropping the bound matches the Delta unsafe-writer path more than the Iceberg nan-count path. a missing bound is not used for skipping. if you'd rather keep the finite bounds and only skip when we know NaN was absent, that needs a nan count (or an unreliable flag) on the stats. I didn't add that here. say if you want that shape instead. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
