Anurag-S1ngh commented on PR #4401: URL: https://github.com/apache/iceberg-python/pull/4401#issuecomment-6957363577
Thanks Fokko, all addressed. - Counts are now added across `write()` calls instead of replaced, with a test that writes two batches to one file. - The statistics plan is computed once in `__init__` and passed into the NaN counter, which skips columns whose metrics mode is `none`. `write.metadata.metrics.default=none` now gives an empty `nan_value_counts`. pyiceberg does not implement `write.metadata.metrics.max-inferred-column-defaults` yet, so there was nothing else to skip. - Dropped the `try/except` and the `count is not None` check. `pc.is_nan` and `pc.sum` do not raise for float or double columns, and `pc.sum` with `min_count=0` always returns a valid scalar. - The counts are passed into `data_file_statistics_from_parquet_metadata` as an optional argument, so the `add_files` path can reuse it later. - Added the tests you asked for and updated the merge manifests integration expectation. `add_files` is out of scope here and #4396 stays open; I noted it in the description. Thanks for the nested fields note, I will follow up with structs counted and fields inside lists and maps left out. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
