jiayuasu opened a new issue, #3283:
URL: https://github.com/apache/sedona/issues/3283
## Expected behavior
```python
client = Client.open("https://earth-search.aws.element84.com/v1";)
items = client.search(
collection_id="sentinel-2-c1-l2a",
datetime="2025",
max_items=100,
)
```
should fetch roughly 100 items from the STAC API and return quickly, the way
pystac-client's `max_items` does.
## Actual behavior
`CollectionClient.load_items_df` only passes `max_items` to the reader as
`itemsLimitMax` when **no** filters are provided. As soon as `bbox` or
`datetime` is given, it falls back to `df.limit(max_items)` applied *behind*
the filters — which cannot bound the scan, since a limit behind predicates is
not pushed to the source. The client also never sets `itemsLimitPerRequest`, so
the scan pages at the default 10 items per request.
The result: the driver sequentially enumerates **every** matching result
page before Spark trims to `max_items`. For the query above (~hundreds of
thousands of matching items for a year of `sentinel-2-c1-l2a`), that is tens of
thousands of sequential HTTP requests — in our integration environment the job
ran for the full 1-hour timeout without completing, for a query that asks for
100 items.
## Proposed fix
Follow pystac-client semantics for the searches the STAC API can represent
directly: when a search against a named collection uses at most one `bbox` and
one `datetime` interval (and no ID or geometry filter) with a positive
`max_items`, send those constraints as parameters of the collection's
advertised `rel=items` endpoint, trust the API's spatial and temporal matching,
and stop enumeration after `max_items` server results (`itemsLimitMax`), with a
sane page size (`min(200, max_items)`).
Multiple bounding boxes or datetime intervals, ID filters, and geometry
filters are Sedona extensions evaluated by Spark; those shapes keep the current
Spark-filtered behavior (fully enumerated, then filtered and limited) because a
raw fetch cap applied before Spark-side predicates could return fewer matching
items than requested.
Note this makes the representable searches API-owned in semantics (e.g. an
interval-valued Item that a conforming server returns for a datetime search is
kept, rather than re-filtered against its nominal `datetime` by Spark) — which
is exactly what pystac-client users expect.
A related fix in the same code path: `_move_attributes_to_properties`
stringifies `datetime.datetime` values with `str()`, producing naive local-time
strings in PyStacItem round-trips; they should be normalized to UTC RFC 3339
(`Z`) strings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
