Fokko commented on code in PR #4401:
URL: https://github.com/apache/iceberg-python/pull/4401#discussion_r4598703783


##########
pyiceberg/io/pyarrow.py:
##########
@@ -2889,6 +2929,7 @@ def write(self, table: pa.Table) -> None:
                 raise
             self._fos = fos
         self._writer.write(table, row_group_size=self._row_group_size)
+        self._nan_value_counts.update(_nan_value_counts_from_arrow(table, 
self._file_schema))

Review Comment:
   `dict.update` replaces the counts from earlier batches instead of adding to 
them, so only the last `write()` call counts:
   
   ```python
   w.write(pa.table({"d": [nan, nan]}))
   w.write(pa.table({"d": [1.0]}))
   w.close().nan_value_counts  # {1: 0}, expected {1: 2}
   ```
   
   This is the dangerous case: a count of `0` makes 
`_InclusiveMetricsEvaluator.visit_is_nan` return `ROWS_CANNOT_MATCH`, so `d IS 
NAN` would skip a file that does contain NaNs. `write_file` only calls 
`write()` once today, but `FileFormatWriter` is meant to be written to 
repeatedly. Could you add the counts together instead, and add a test that 
calls `write()` more than once?
   
   ```python
   for field_id, count in _nan_value_counts_from_arrow(table, 
self._file_schema).items():
       self._nan_value_counts[field_id] = self._nan_value_counts.get(field_id, 
0) + count
   ```



##########
pyiceberg/io/pyarrow.py:
##########
@@ -2897,10 +2938,13 @@ def close(self) -> DataFileStatistics:
             raise ValueError("Cannot close a writer that was never written to")
         with self._fos:
             self._writer.close()
-            self._result = data_file_statistics_from_parquet_metadata(
-                parquet_metadata=self._writer.writer.metadata,
-                stats_columns=compute_statistics_plan(self._file_schema, 
self._properties),
-                
parquet_column_mapping=parquet_path_to_id_mapping(self._file_schema),
+            self._result = replace(

Review Comment:
   Using `replace` works, but I'd rather pass the NaN counts into 
`data_file_statistics_from_parquet_metadata` as an optional argument. That 
keeps all the statistics logic in one place and makes it easier to reuse for 
the `add_files` path later.



##########
pyiceberg/io/pyarrow.py:
##########
@@ -2717,6 +2717,45 @@ def parquet_path_to_id_mapping(
     return result
 
 
+def _nan_value_counts_from_arrow(table: pa.Table, schema: Schema) -> dict[int, 
int]:
+    """Count the non-null NaN values of the top-level float and double columns.
+
+    PyArrow does not expose the NaN count of a Parquet column statistic, so 
the counts are taken
+    from the data before it is written. Columns that cannot be counted are 
left out, which only
+    means the file cannot be pruned on NaN counts.
+
+    Args:
+        table (pyarrow.Table): The data that is written to the file.
+        schema (Schema): The schema of the file.
+
+    Returns:
+        A mapping of Iceberg field ID to the number of non-null NaN values.
+    """
+    nan_value_counts: dict[int, int] = {}
+    for pos, arrow_field in enumerate(table.schema):

Review Comment:
   This doesn't look at the metrics mode, so a column set to `none` still gets 
a NaN count, while its value and null counts are left out:
   
   ```
   write.metadata.metrics.default=none
   → value_counts={}, null_value_counts={}, nan_value_counts={1: 1}
   ```
   
   Java skips these columns in `MetricsUtil.createNanValueCounts`, using 
`metricsConfig.columnMode(id) != MetricsModes.None.get()`. The same applies to 
columns past `write.metadata.metrics.max-inferred-column-defaults`. Could you 
pass the result of `compute_statistics_plan(...)` in and skip fields whose mode 
is `NONE`? You could compute the plan once in `__init__` and reuse it in 
`close()` as well.



##########
pyiceberg/io/pyarrow.py:
##########
@@ -2717,6 +2717,45 @@ def parquet_path_to_id_mapping(
     return result
 
 
+def _nan_value_counts_from_arrow(table: pa.Table, schema: Schema) -> dict[int, 
int]:
+    """Count the non-null NaN values of the top-level float and double columns.
+
+    PyArrow does not expose the NaN count of a Parquet column statistic, so 
the counts are taken
+    from the data before it is written. Columns that cannot be counted are 
left out, which only
+    means the file cannot be pruned on NaN counts.
+
+    Args:
+        table (pyarrow.Table): The data that is written to the file.
+        schema (Schema): The schema of the file.
+
+    Returns:
+        A mapping of Iceberg field ID to the number of non-null NaN values.
+    """
+    nan_value_counts: dict[int, int] = {}
+    for pos, arrow_field in enumerate(table.schema):
+        if (field_id := _get_field_id(arrow_field)) is None:
+            continue
+
+        try:
+            field_type = schema.find_field(field_id).field_type
+        except ValueError:
+            continue
+
+        if not isinstance(field_type, (FloatType, DoubleType)):
+            continue
+
+        try:
+            count = pc.sum(pc.is_nan(table.column(pos)), min_count=0)
+        except (pyarrow.lib.ArrowInvalid, 
pyarrow.lib.ArrowNotImplementedError) as e:
+            logger.warning("Could not count NaN values for column %s: %s", 
arrow_field.name, e)
+            continue
+
+        if count is not None:

Review Comment:
   Nit: `pc.sum` always returns a scalar, and with `min_count=0` its value is 
never null, so this check is always true.



##########
tests/io/test_pyarrow.py:
##########
@@ -3596,6 +3596,39 @@ def test_append_to_table_bucketed_by_time(catalog: 
InMemoryCatalog) -> None:
     assert table.scan(row_filter=EqualTo("event_time", 
"22:31:08")).to_arrow()["id"].to_pylist() == [2]
 
 
+def test_append_counts_nan_values(catalog: InMemoryCatalog) -> None:

Review Comment:
   Could we also add tests that (1) call `ParquetFormatWriter.write()` several 
times on the same file and check that the counts are added up, and (2) check 
that a column with metrics mode `none` is left out of `nan_value_counts`?
   
   A follow-up for nested fields sounds good. For reference, Java leaves out 
fields inside lists and maps (`inMapOrList`) but does count floats and doubles 
nested in structs.



##########
pyiceberg/io/pyarrow.py:
##########
@@ -2717,6 +2717,45 @@ def parquet_path_to_id_mapping(
     return result
 
 
+def _nan_value_counts_from_arrow(table: pa.Table, schema: Schema) -> dict[int, 
int]:
+    """Count the non-null NaN values of the top-level float and double columns.
+
+    PyArrow does not expose the NaN count of a Parquet column statistic, so 
the counts are taken
+    from the data before it is written. Columns that cannot be counted are 
left out, which only
+    means the file cannot be pruned on NaN counts.
+
+    Args:
+        table (pyarrow.Table): The data that is written to the file.
+        schema (Schema): The schema of the file.
+
+    Returns:
+        A mapping of Iceberg field ID to the number of non-null NaN values.
+    """
+    nan_value_counts: dict[int, int] = {}
+    for pos, arrow_field in enumerate(table.schema):
+        if (field_id := _get_field_id(arrow_field)) is None:
+            continue
+
+        try:
+            field_type = schema.find_field(field_id).field_type
+        except ValueError:
+            continue
+
+        if not isinstance(field_type, (FloatType, DoubleType)):
+            continue
+
+        try:
+            count = pc.sum(pc.is_nan(table.column(pos)), min_count=0)
+        except (pyarrow.lib.ArrowInvalid, 
pyarrow.lib.ArrowNotImplementedError) as e:

Review Comment:
   Nit: can this actually raise for a float or double column? If not, I'd drop 
the `try/except`. If it can, logging a warning on every `write()` could get 
noisy.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to