jiayuasu opened a new pull request, #3284: URL: https://github.com/apache/sedona/pull/3284
## Did you read the Contributor Guide? - Yes, I have read the [Contributor Rules](https://sedona.apache.org/latest/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/latest/community/develop/) ## Is this PR related to a ticket? - Yes, and the PR name follows the format `[GH-XXX] my subject`. Closes #3283 ## What changes were proposed in this PR? `CollectionClient.load_items_df` only passed `max_items` to the reader as `itemsLimitMax` when no filters were provided. With a `bbox` or `datetime` filter it fell back to `df.limit(max_items)` behind the filters — which cannot bound the scan — so `client.search(collection_id="sentinel-2-c1-l2a", datetime="2025", max_items=100)` sequentially enumerated every matching result page (hundreds of thousands of items at 10 per request) before Spark trimmed to 100. In our integration environment this ran to a 1-hour timeout. Searches the STAC API can represent directly now follow **pystac-client semantics**: - When a search against a named collection uses at most one `bbox` and one `datetime` interval, no ID or geometry filter, and a positive `max_items`, the Python client forwards the constraints through internal reader options. - The datasource applies them as parameters of the collection's advertised JSON `rel=items` endpoint (preserving custom, cross-origin, and query-bearing hrefs), trusts the conforming API's spatial and temporal matching, and stops enumeration after `max_items` server results via the existing counting walk, paging at `min(200, max_items)`. The motivating search becomes ~1 request. - Endpoints that already own a same-kind constraint (`bbox`/`bbox-crs`/`intersects`/`datetime`, matched on percent-decoded parameter names) are rejected rather than widened or duplicated, and this mode refuses to combine with Spark-pushed predicates, whose retained Filter would discard rows the cap already counted. - Multiple bounding boxes or datetime intervals, ID filters, and geometry filters are Sedona extensions evaluated by Spark; those shapes keep the current behavior (full enumeration, then Spark filters and the final limit) because a raw fetch cap before Spark-side predicates could return fewer matching items than requested. Note this makes the representable searches API-owned in semantics: e.g. an interval-valued Item a conforming server returns for a datetime search is kept rather than re-filtered against its nominal `datetime` — which matches what pystac-client users expect. Also fixes a timezone bug in `_move_attributes_to_properties`: PySpark materializes timestamps in the local timezone, so item round-trips previously produced naive local-time strings; datetime values are now normalized to UTC RFC 3339 (`Z`) strings. ## How was this patch tested? - New Scala tests: the API-mode request chain against a local HTTP server (cursor pagination stops at the cap, child links never fetched, custom query-bearing href preserved) in `StacBatchTest`, plus `applyClientApiSearchParameters` unit tests (encoding, fragment placement, conflicting-constraint rejection incl. percent-encoded names) in `StacBatchUrlTest`. All STAC suites pass under `-Dspark=3.5`: 73 passed. - New Python tests: API-search option matrix (single vs multiple bbox/datetime, geometry/ids fallback, URL validation incl. antimeridian bbox), production reader-option wiring, Spark-filtered fallback, and UTC round-trip of interval item datetimes. 31 passed. ## Did this PR include necessary documentation updates? - Yes, I have updated the documentation. The STAC tutorial (EN/ZH) now documents the pystac-client `max_items` semantics and which search shapes are API-owned vs Spark-evaluated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
