jiayuasu opened a new issue, #3277: URL: https://github.com/apache/sedona/issues/3277
## Expected behavior When a STAC items or search endpoint already contains a datetime query parameter, that parameter defines the source relation. Pushing a Spark temporal predicate must not add a second datetime parameter. A safe fallback is to preserve the endpoint parameter and let Spark's retained residual predicate apply the pushed condition. ## Actual behavior StacBatch.getItemLink preserves the endpoint query and then StacUtils.addFiltersToUrl unconditionally appends the pushed temporal filter. On Sedona 1.9.1 and current master, this produces: ~~~text ...?datetime=<endpoint-range>&limit=10&datetime=<pushed-range> ~~~ Repeated scalar parameters are handled differently across STAC servers. A server may reject the request or honor only the first or last value. If it honors the pushed value and ignores the endpoint value, rows outside the source relation can survive Spark's residual filter. The current call chain is: - [StacBatch.getItemLink](https://github.com/apache/sedona/blob/ccee236e00184438708c4e88be59b47002759bea/spark/common/src/main/scala/org/apache/spark/sql/sedona_sql/io/stac/StacBatch.scala#L290-L320) preserves the endpoint query and calls addFiltersToUrl. - [StacUtils.addFiltersToUrl](https://github.com/apache/sedona/blob/ccee236e00184438708c4e88be59b47002759bea/spark/common/src/main/scala/org/apache/spark/sql/sedona_sql/io/stac/StacUtils.scala#L477-L486) appends another datetime whenever a temporal filter is present. ## Steps to reproduce ~~~scala import java.time.LocalDateTime import org.apache.spark.sql.execution.datasource.stac.TemporalFilter import org.apache.spark.sql.sedona_sql.io.stac.StacBatch import org.apache.spark.sql.types.StructType val batch = StacBatch( null, "", """{"links":[]}""", StructType(Nil), Map.empty, None, None, None) val pushedFilter = Some( TemporalFilter.GreaterThanFilter( "datetime", LocalDateTime.parse("2025-03-06T00:00:00"))) val result = batch.getItemLink( "https://example.test/items"; + "?datetime=2025-02-01T00:00:00Z/2025-02-28T23:59:59Z", 10, None, pushedFilter) println(result) ~~~ Actual output: ~~~text https://example.test/items?datetime=2025-02-01T00:00:00Z/2025-02-28T23:59:59Z&limit=10&datetime=2025-03-06T00:00:00.000000000Z/.. ~~~ The endpoint range and pushed range are disjoint, so the correct result is empty. A server that honors only the final parameter can instead return March items, which satisfy the Spark residual predicate but violate the endpoint's February constraint. ## Affected versions - Sedona 1.7.2 through 1.9.1 - Current master at ccee236e In versions 1.7.2 through 1.9.0, existing queries are additionally followed by another ?limit=, producing a malformed form such as: ~~~text ...?datetime=<endpoint-range>?limit=10&datetime=<pushed-range> ~~~ Sedona 1.7.1 predates this filter-pushdown path. ## Impact Depending on server behavior, scans can fail, lose remote pruning, or return rows outside the endpoint-defined source relation. ## Suggested fix Before appending the pushed temporal filter: 1. Parse query parameters separately from the fragment and recognize percent-encoded parameter names. 2. If the endpoint already owns any datetime parameter, preserve it unchanged and do not append another one. 3. Let Spark's retained temporal predicate enforce the pushed condition. Optionally, exactly one valid endpoint interval may be intersected with the pushed interval and serialized as one parameter. Repeated, empty, reversed, or unparsable endpoint values should remain unchanged rather than being removed or replaced. Suggested regression cases: - a valid endpoint interval plus a pushed predicate; - malformed, empty, and repeated endpoint datetime values; - existing query parameters and URL fragments; - percent-encoded parameter names. ## Sedona version 1.9.1 and current master ## Apache Spark version N/A; reproduces directly in shared URL construction ## API type Scala ## Scala version 2.12 and 2.13 ## JRE version 11+ ## Environment Local unit-level reproduction; runtime behavior affects any Spark STAC scan whose endpoint URL already contains datetime. ## Existing issues - [x] I searched the existing issues and did not find a duplicate. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
