FurkhanShaikh commented on issue #4079:
URL:
https://github.com/apache/iceberg-python/issues/4079#issuecomment-7054884613
On the checklist item **"Make sure the metrics evaluators don't read
spec-encoded bounds (x, y[, z, m]) as WKB"**: I looked into this, and I think
the evaluators are already safe on `main`. Along the way I found a related
crash in `inspect`.
**Metrics evaluators: no path reads geo bounds today**
- No literal type in `pyiceberg/expressions/literals.py` converts to
`geometry`/`geography`. So `EqualTo`, `NotEqualTo`,
`LessThan`/`LessThanOrEqual`, `GreaterThan`/`GreaterThanOrEqual`, `In`,
`NotIn`, `StartsWith` and `NotStartsWith` on a geo column all fail at bind time
with `TypeError: Cannot convert BinaryLiteral into geometry`. `BindVisitor`
rejects predicates that are already bound.
- `IsNaN`/`NotNaN` bind to `AlwaysFalse`/`AlwaysTrue`, and
`IsNull`/`NotNull` only use value/null counts.
- End to end with the integration image (Spark 4.1.3, Iceberg 1.12.0), Spark
writes **no** lower/upper bounds for geometry/geography columns. So there are
currently no geo bounds to misread.
So nothing needs fixing in
`_InclusiveMetricsEvaluator`/`_StrictMetricsEvaluator` right now. The item is
still worth keeping in mind for spatial predicates (e.g. `st_intersects`),
since those would be the first to decode the `x:y[:z:m]` bounds.
**`inspect.files()` crashes on geo tables when `geoarrow-pyarrow` is
installed**
Repro: with `spark.sql.geospatial.enabled=true`, Spark creates a V3 table
`(id BIGINT, geom GEOMETRY(4326), geog GEOGRAPHY(4326))` and inserts three
points plus a NULL row. PyIceberg then loads the table and calls
`table.inspect.files()`:
```
File "pyiceberg/table/inspect.py", line 790, in _get_files_from_manifest
return pa.Table.from_pylist(
...
pyarrow.lib.ArrowNotImplementedError: extension
```
`_get_files_schema` types the `readable_metrics` `lower_bound`/`upper_bound`
fields with `schema_to_pyarrow(field_type)` (`inspect.py:816-817`). For geo
columns that is the `geoarrow.wkb` extension type, and `pa.Table.from_pylist`
can't build it from Python values. This fails even though every bound is
`None`. Without geoarrow installed, the type falls back to `large_binary` and
`files()` works. `entries()` builds the same struct (`inspect.py:178-179`), so
it is probably affected too, but I haven't run it.
When bounds are present, they are spec-encoded `x:y[:z:m]` points rather
than WKB, so labelling them `geoarrow.wkb` would be wrong anyway. Plain binary
for geo bound fields looks like the simplest fix. Would that work, or would you
rather decode them into something more readable? I'm happy to open a PR either
way.
Note: on `main`, loading a Spark-written geo table currently fails earlier,
with `ValidationError: Could not parse geometry(OGC:CRS84) into a GeometryType`
(item 1), and a full scan fails with `ResolveError: Cannot promote an binary to
geometry`. For the repro I patched the type regexes locally to accept the
unquoted CRS. #4422 covers loading and doesn't touch `inspect.py`, so I'd build
any fix on top of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]