jiayuasu opened a new pull request, #3291: URL: https://github.com/apache/sedona/pull/3291
## Did you read the Contributor Guide? - Yes, I have read the [Contributor Rules](https://sedona.apache.org/latest/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/latest/community/develop/) ## Is this PR related to a ticket? - Yes, and the PR name follows the format `[GH-XXX] my subject`. Closes #3290 ## What changes were proposed in this PR? `_api_search_options` opened with `max_items is None or max_items <= 0 → return None`, so the API-native search path added in #3287 was only taken when the caller also bounded the result set. `save_to_geoparquet` structurally cannot supply `max_items` — it neither accepts nor forwards one — so its single bbox and datetime interval always fell into the uncapped fallback: the reader enumerated the collection with no bbox on the request and Spark discarded the non-matching Items afterwards. `max_items` is a result cap, not a precondition for search. This PR separates the two: - the Python guard now rejects only a non-positive `max_items` (a request for nothing, not a request for everything); a representable predicate is pushed to the Items endpoint whether or not a cap was supplied - an uncapped API search sets the client's 200-Item page size, so pagination does not fall back to the datasource default of 10 - `StacBatch` no longer requires a positive `itemsLimitMax` for client API search options: an uncapped search walks the API's own pagination chain to the end, exactly like every other uncapped scan; a positive cap keeps the counted walk unchanged - the incompatibility with Spark-pushed filters is still rejected ## How was this patch tested? - Two new Python unit tests: the options for the unbounded representable shape, and the full `load_items_df` wiring for the shape `save_to_geoparquet` issues (search options present, no `itemsLimitMax`, `itemsLimitPerRequest=200`, no `df.limit`, no Spark re-filtering). Both fail without the source change. - One new `StacBatchTest` case driving `collectItemLinks` with search options and `needCountNextItems=false` against a live local HTTP server with a finite pagination chain: fails on the previous `require` with `IllegalArgumentException: requirement failed: Client API search options require a positive itemsLimitMax`, passes with the fix. - `python/tests/stac/test_collection_client.py`: 35 passed. `StacBatchTest` + `StacBatchUrlTest` + `StacDataSourceTest`: 32 passed, 0 failed (Spark 4.1, Scala 2.13, JDK 17). - Verified end to end against the live Element84 Earth Search API: the reproducer in #3290 (a January-2025 bbox export from `sentinel-2-c1-l2a`) drops from tens of minutes of collection enumeration to seconds, writing 1,753 Items. ## Did this PR include necessary documentation updates? - Yes, I have updated the documentation. (`docs/tutorial/files/stac-sedona-spark.md` and `.zh.md` now document the unbounded case.) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
