jiayuasu opened a new issue, #3286:
URL: https://github.com/apache/sedona/issues/3286
## Expected behavior
```python
client = Client.open("https://earth-search.aws.element84.com/v1";)
items = client.search(
collection_id="sentinel-2-c1-l2a",
bbox=[[-10.0, 35.0, 5.0, 45.0], [100.0, -10.0, 120.0, 10.0]],
datetime="2025",
max_items=100,
)
```
should fetch roughly `max_items` results per bounding box from the STAC API
and return quickly, the way a bounded search is expected to behave.
## Actual behavior
GH-3283 added API-native capped searches, but only for at most **one** bbox.
A search with multiple bounding boxes takes the Spark-filtered fallback: the
driver sequentially enumerates **every** result page of the collection before
Spark applies the bbox predicates and the final limit. The fallback also
inherits the datasource's default page size of 10 Items per request, so a
bounded multi-bbox query can issue tens of thousands of sequential HTTP
requests and run into service throttling.
## Proposed fix
Each bbox maps directly to a STAC API Collection Items request, and Sedona's
list-of-bboxes extension represents their union, so a bounded multi-bbox search
can be decomposed instead of falling back:
- For a named HTTP collection with a positive `max_items`, one or more valid
bboxes, at most one `datetime` interval, and no ID or geometry filter: issue
one API-native capped search per bbox (`itemsLimitMax=max_items`,
`itemsLimitPerRequest=min(200, max_items)`), union the per-bbox DataFrames,
deduplicate by `(collection, id)`, and apply one final global `max_items` limit.
- Each request may safely use the full `max_items` cap: if any bbox alone
contains at least that many Items it can satisfy the limit; otherwise every
bbox is fully represented, so deduplicate-then-limit yields the requested
global result size.
- Shapes that still require a Spark-side predicate (multiple datetime
intervals, geometry filters, IDs, or an invalid bbox) keep the uncapped
fallback so a raw fetch cap cannot starve residual predicates — but use
200-Item request pages to reduce pagination overhead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
