FurkhanShaikh commented on issue #4079:
URL: 
https://github.com/apache/iceberg-python/issues/4079#issuecomment-7054884613

   On the checklist item **"Make sure the metrics evaluators don't read 
spec-encoded bounds (x, y[, z, m]) as WKB"**: I looked into this, and I think 
the evaluators are already safe on `main`. Along the way I found a related 
crash in `inspect`.
   
   **Metrics evaluators: no path reads geo bounds today**
   
   - No literal type in `pyiceberg/expressions/literals.py` converts to 
`geometry`/`geography`. So `EqualTo`, `NotEqualTo`, 
`LessThan`/`LessThanOrEqual`, `GreaterThan`/`GreaterThanOrEqual`, `In`, 
`NotIn`, `StartsWith` and `NotStartsWith` on a geo column all fail at bind time 
with `TypeError: Cannot convert BinaryLiteral into geometry`. `BindVisitor` 
rejects predicates that are already bound.
   - `IsNaN`/`NotNaN` bind to `AlwaysFalse`/`AlwaysTrue`, and 
`IsNull`/`NotNull` only use value/null counts.
   - End to end with the integration image (Spark 4.1.3, Iceberg 1.12.0), Spark 
writes **no** lower/upper bounds for geometry/geography columns. So there are 
currently no geo bounds to misread.
   
   So nothing needs fixing in 
`_InclusiveMetricsEvaluator`/`_StrictMetricsEvaluator` right now. The item is 
still worth keeping in mind for spatial predicates (e.g. `st_intersects`), 
since those would be the first to decode the `x:y[:z:m]` bounds.
   
   **`inspect.files()` crashes on geo tables when `geoarrow-pyarrow` is 
installed**
   
   Repro: with `spark.sql.geospatial.enabled=true`, Spark creates a V3 table 
`(id BIGINT, geom GEOMETRY(4326), geog GEOGRAPHY(4326))` and inserts three 
points plus a NULL row. PyIceberg then loads the table and calls 
`table.inspect.files()`:
   
   ```
     File "pyiceberg/table/inspect.py", line 790, in _get_files_from_manifest
       return pa.Table.from_pylist(
     ...
   pyarrow.lib.ArrowNotImplementedError: extension
   ```
   
   `_get_files_schema` types the `readable_metrics` `lower_bound`/`upper_bound` 
fields with `schema_to_pyarrow(field_type)` (`inspect.py:816-817`). For geo 
columns that is the `geoarrow.wkb` extension type, and `pa.Table.from_pylist` 
can't build it from Python values. This fails even though every bound is 
`None`. Without geoarrow installed, the type falls back to `large_binary` and 
`files()` works. `entries()` builds the same struct (`inspect.py:178-179`), so 
it is probably affected too, but I haven't run it.
   
   When bounds are present, they are spec-encoded `x:y[:z:m]` points rather 
than WKB, so labelling them `geoarrow.wkb` would be wrong anyway. Plain binary 
for geo bound fields looks like the simplest fix. Would that work, or would you 
rather decode them into something more readable? I'm happy to open a PR either 
way.
   
   Note: on `main`, loading a Spark-written geo table currently fails earlier, 
with `ValidationError: Could not parse geometry(OGC:CRS84) into a GeometryType` 
(item 1), and a full scan fails with `ResolveError: Cannot promote an binary to 
geometry`. For the repro I patched the type regexes locally to accept the 
unquoted CRS. #4422 covers loading and doesn't touch `inspect.py`, so I'd build 
any fix on top of it.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to