Copilot commented on code in PR #3116: URL: https://github.com/apache/sedona/pull/3116#discussion_r3601790055
########## docs/blog/posts/stac-catalog-reader.md: ########## @@ -0,0 +1,214 @@ +--- +date: + created: 2026-07-17 +links: + - STAC reader tutorial: https://sedona.apache.org/latest/tutorial/files/stac-sedona-spark/ + - STAC specification: https://stacspec.org/ + - earth-search (public Sentinel-2 STAC): https://earth-search.aws.element84.com/v1 +authors: + - jia +title: "Join the Sky to the Ground: Spatial Joins over STAC Catalogs" +--- + +# Join the Sky to the Ground: Spatial Joins over STAC Catalogs + +You have the shapes. Somewhere in a petabyte of satellite imagery are exactly the scenes that cover them. The hard part was never the analysis — it was the plumbing in between. + + + +<!-- more --> + +Every few days, satellites re-photograph the entire planet. A [**STAC catalog**](https://stacspec.org/) (SpatioTemporal Asset Catalog) is how that firehose is published — Sentinel-2, Landsat, NAIP, Maxar, and Microsoft's Planetary Computer all speak it. Each *item* in a catalog describes one captured scene: its footprint geometry, its timestamp, its cloud cover, and links to the actual pixels. + +SedonaSpark's STAC reader parses all of that into a Sedona DataFrame whose `geometry` column is a real Sedona geometry — which means the catalog drops straight into a spatial join with the vector shapes you already have. This post walks the whole path, from a live catalog read to the spatial join, with real output throughout. + +## One line to a Sedona DataFrame + +Point the `stac` format at any STAC endpoint — a public API, an `s3a://` object, or a local `collection.json`. Here we open Element 84's public `earth-search` catalog and pull the live Sentinel-2 archive: + +```python +from sedona.spark import SedonaContext + +sedona = SedonaContext.create(SedonaContext.builder().master("local[*]").getOrCreate()) + +base = "https://earth-search.aws.element84.com/v1"; +scenes = sedona.read.format("stac").load(f"{base}/collections/sentinel-2-l2a") + +scenes.printSchema() +``` + +The reader gives every STAC field a typed column. The important ones: `geometry` arrives as a native geometry, `datetime` as a timestamp, and the [EO extension](https://github.com/stac-extensions/eo)'s `eo:cloud_cover` is lifted to a top-level double. + +``` + |-- id: string + |-- bbox: array<double> + |-- geometry: geometry <- a real Sedona geometry + |-- datetime: timestamp + |-- eo:cloud_cover: double + |-- eo:snow_cover: double + |-- platform: string + |-- constellation: string + |-- collection: string + |-- assets: map<string, struct<href, type, title, roles>> +``` + +A peek at the rows — a live descending pass captured the day this ran: + +```python +scenes.selectExpr( + "id", + "datetime", + "`eo:cloud_cover` AS cloud", + "ST_GeometryType(geometry) AS shape", +).show(3, truncate=False) +``` + +``` ++--------------------------+-------------------------+------+-----------+ +|id |datetime |cloud |shape | ++--------------------------+-------------------------+------+-----------+ +|S2B_50LQK_20260717_0_L2A |2026-07-17 02:31:27.091 |0.0 |ST_Polygon | +|S2B_50LPJ_20260717_0_L2A |2026-07-17 02:31:39.810 |1.55 |ST_Polygon | +|S2B_50LQL_20260717_0_L2A |2026-07-17 02:31:16.128 |6.95 |ST_Polygon | ++--------------------------+-------------------------+------+-----------+ +``` + +Real footprints, real cloud cover, real timestamps — with nothing more than a format string. + +## The API does the filtering + +The Sentinel-2 archive is tens of millions of scenes. You never want all of them. When you filter on **space**, **time**, or **cloud cover**, Sedona pushes those predicates *down to the STAC API itself* — only the matching tiles ever cross the network. Apply the spatial predicate *on the read* (not buried in a join) and it becomes an API-side bounding query. + + + +```python +sf = ( + "POLYGON((-122.52 37.70, -122.36 37.70, " + "-122.36 37.83, -122.52 37.83, -122.52 37.70))" +) + +sf_scenes = ( + sedona.read.format("stac") + .load(f"{base}/collections/sentinel-2-l2a") + # pushed to the STAC /search API as a bbox query + .filter(f"ST_Intersects(ST_GeomFromText('{sf}'), geometry)") + .filter("`eo:cloud_cover` < 20") +) Review Comment: This section and the figure describe spatial + temporal + cloud-cover predicate pushdown, but the code sample only applies spatial and cloud filters. Adding a `datetime` predicate here would make the example match the narrative and demonstrate temporal pushdown explicitly. ########## docs/blog/posts/stac-catalog-reader.md: ########## @@ -0,0 +1,214 @@ +--- +date: + created: 2026-07-17 +links: + - STAC reader tutorial: https://sedona.apache.org/latest/tutorial/files/stac-sedona-spark/ + - STAC specification: https://stacspec.org/ + - earth-search (public Sentinel-2 STAC): https://earth-search.aws.element84.com/v1 +authors: + - jia +title: "Join the Sky to the Ground: Spatial Joins over STAC Catalogs" +--- + +# Join the Sky to the Ground: Spatial Joins over STAC Catalogs + +You have the shapes. Somewhere in a petabyte of satellite imagery are exactly the scenes that cover them. The hard part was never the analysis — it was the plumbing in between. + + + +<!-- more --> + +Every few days, satellites re-photograph the entire planet. A [**STAC catalog**](https://stacspec.org/) (SpatioTemporal Asset Catalog) is how that firehose is published — Sentinel-2, Landsat, NAIP, Maxar, and Microsoft's Planetary Computer all speak it. Each *item* in a catalog describes one captured scene: its footprint geometry, its timestamp, its cloud cover, and links to the actual pixels. + +SedonaSpark's STAC reader parses all of that into a Sedona DataFrame whose `geometry` column is a real Sedona geometry — which means the catalog drops straight into a spatial join with the vector shapes you already have. This post walks the whole path, from a live catalog read to the spatial join, with real output throughout. + +## One line to a Sedona DataFrame + +Point the `stac` format at any STAC endpoint — a public API, an `s3a://` object, or a local `collection.json`. Here we open Element 84's public `earth-search` catalog and pull the live Sentinel-2 archive: + +```python +from sedona.spark import SedonaContext + +sedona = SedonaContext.create(SedonaContext.builder().master("local[*]").getOrCreate()) + +base = "https://earth-search.aws.element84.com/v1"; +scenes = sedona.read.format("stac").load(f"{base}/collections/sentinel-2-l2a") + +scenes.printSchema() +``` + +The reader gives every STAC field a typed column. The important ones: `geometry` arrives as a native geometry, `datetime` as a timestamp, and the [EO extension](https://github.com/stac-extensions/eo)'s `eo:cloud_cover` is lifted to a top-level double. + +``` + |-- id: string + |-- bbox: array<double> + |-- geometry: geometry <- a real Sedona geometry + |-- datetime: timestamp + |-- eo:cloud_cover: double + |-- eo:snow_cover: double + |-- platform: string + |-- constellation: string + |-- collection: string + |-- assets: map<string, struct<href, type, title, roles>> +``` + +A peek at the rows — a live descending pass captured the day this ran: + +```python +scenes.selectExpr( + "id", + "datetime", + "`eo:cloud_cover` AS cloud", + "ST_GeometryType(geometry) AS shape", +).show(3, truncate=False) +``` + +``` ++--------------------------+-------------------------+------+-----------+ +|id |datetime |cloud |shape | ++--------------------------+-------------------------+------+-----------+ +|S2B_50LQK_20260717_0_L2A |2026-07-17 02:31:27.091 |0.0 |ST_Polygon | +|S2B_50LPJ_20260717_0_L2A |2026-07-17 02:31:39.810 |1.55 |ST_Polygon | +|S2B_50LQL_20260717_0_L2A |2026-07-17 02:31:16.128 |6.95 |ST_Polygon | ++--------------------------+-------------------------+------+-----------+ +``` + +Real footprints, real cloud cover, real timestamps — with nothing more than a format string. + +## The API does the filtering + +The Sentinel-2 archive is tens of millions of scenes. You never want all of them. When you filter on **space**, **time**, or **cloud cover**, Sedona pushes those predicates *down to the STAC API itself* — only the matching tiles ever cross the network. Apply the spatial predicate *on the read* (not buried in a join) and it becomes an API-side bounding query. + + + +```python +sf = ( + "POLYGON((-122.52 37.70, -122.36 37.70, " + "-122.36 37.83, -122.52 37.83, -122.52 37.70))" +) + +sf_scenes = ( + sedona.read.format("stac") + .load(f"{base}/collections/sentinel-2-l2a") + # pushed to the STAC /search API as a bbox query + .filter(f"ST_Intersects(ST_GeomFromText('{sf}'), geometry)") + .filter("`eo:cloud_cover` < 20") +) + +sf_scenes.selectExpr( + "id", "date(datetime) AS day", "round(`eo:cloud_cover`, 1) AS cloud" +).orderBy("day").show(4) +``` + +Only tile `10SEG` — the one covering San Francisco — comes back, at sub-1% cloud: + +``` ++--------------------------+------------+------+ +|id |day |cloud | ++--------------------------+------------+------+ +|S2C_10SEG_20260513_0_L2A |2026-05-13 |0.9 | +|S2A_10SEG_20260515_0_L2A |2026-05-15 |0.1 | +|S2B_10SEG_20260518_0_L2A |2026-05-18 |0.7 | +|S2A_10SEG_20260604_0_L2A |2026-06-04 |0.2 | ++--------------------------+------------+------+ +``` + +The archive's other tens of millions of scenes never touched the wire. + +## Join footprints to your vectors + +This is the payoff. Because `geometry` is a real geometry column, the catalog is just another spatial table — and it joins against your vector layer with the same `ST_Intersects` predicate you'd use anywhere else in Sedona. Your areas of interest on the left, the imagery footprints on the right, and the join is the overlap. + + + +Let's make it concrete and reproducible. Sedona ships a tiny sample STAC collection in its test resources, so you can run this offline, verbatim. We load those scene footprints, define three analyst-drawn AOIs as a vector layer, and ask: **which scenes cover each AOI, and how much of the AOI do they cover?** + +```python +# 1 — the imagery layer (Sedona's bundled sample STAC collection) +scenes = sedona.read.format("stac").load("datasource_stac/collection.json") Review Comment: The offline example currently loads `datasource_stac/collection.json` and claims it is “bundled” with Sedona, but the only copy in this repo is under `spark/common/src/test/resources/...` (test resources), which won’t exist for typical Sedona installs. This makes the snippet non-reproducible as written; point readers at the actual path (Sedona source checkout) or otherwise clarify where to get the file. ########## docs-overrides/hooks/blog_star_cta.py: ########## @@ -0,0 +1,41 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Appends a "Star us on GitHub" call-to-action to the end of every blog post. +# Implemented as an on_page_markdown hook so it applies to all current and +# future posts automatically, with no per-file edits. + +from mkdocs import plugins + +STAR_CTA = """ + +<div class="sedona-star-cta" style="margin-top:3rem;padding:1.5rem 1.5rem 1.75rem;border:1px solid var(--md-default-fg-color--lightest);border-radius:.6rem;background:var(--md-default-fg-color--lightest)"> + <p style="font-weight:700;font-size:1.1rem;margin:0 0 .25rem">⭐ Enjoyed this? Star Apache Sedona on GitHub</p> + <p style="margin:0 0 1rem;color:var(--md-default-fg-color--light)">A star takes two seconds and helps others discover the projects.</p> + <a class="md-button md-button--primary" href="https://github.com/apache/sedona"; target="_blank" rel="noopener">⭐ apache/sedona</a> + <a class="md-button" href="https://github.com/apache/sedona-db"; target="_blank" rel="noopener">⭐ apache/sedona-db</a> + <a class="md-button" href="https://github.com/apache/sedona-spatialbench"; target="_blank" rel="noopener">⭐ apache/sedona-spatialbench</a> +</div> +""" Review Comment: The CTA is injected as raw HTML with extensive inline styles. Since `admonition` + `attr_list` are enabled in `mkdocs.yml`, you can generate the same CTA as Markdown (buttons + admonition) and avoid inline CSS/HTML, which is easier to maintain and less likely to break with theme changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
