james-willis opened a new pull request, #1206:
URL: https://github.com/apache/sedona-db/pull/1206
## What
Wires the `integration/spark-parity` pytest suite (added in #1159) into the
`python` workflow, answering the reviewer question on #1203: "Are they running
in CI now?" — with this PR, yes.
Changes, all in `.github/workflows/python.yml` plus a one-sentence README
update:
- `integration/spark-parity/**` added to the `pull_request` paths filter.
- New steps appended to the existing `test` job, after the current test
steps (the job already builds and installs sedonadb editable, which the suite
imports):
- `actions/setup-java@v6` (temurin 17) — the JVM Sedona Spark needs.
- `pip install "pyspark>=4.0" apache-sedona` — the compatibility target.
- An `actions/cache@v6` step for the Ivy jar directory
(`~/.ivy2-spark-parity`).
- `python -m pytest -vv` run from `integration/spark-parity`, with
`SEDONADB_SPARK_IVY_DIR` pointed at the cached directory.
- `integration/spark-parity/README.md`: the "not wired into CI" sentence now
says the suite runs in the python workflow.
## Cache design
The cache key is `spark-parity-jars-${{ runner.os }}-${{
hashFiles('python/sedonadb/python/sedonadb/testing_spark.py') }}`.
`testing_spark.py` is the file that pins `SEDONA_SPARK_VERSION` and
`GEOTOOLS_WRAPPER_VERSION` — the exact Maven coordinates Ivy resolves — so the
cache invalidates precisely when the jar pins change and never goes stale on a
version bump. `SEDONADB_SPARK_IVY_DIR` is set to the same path the cache step
saves, so warm runs skip the Maven download entirely.
## Relationship to #1159 and #1203
The suite on main is currently the small #1159 version (`test_rs_scalar.py`,
`test_rs_raster_out.py`). #1203 expands the suite independently; this PR is
deliberately **not** stacked on it — whichever merges second simply runs
more/newer tests through the same lane.
## Lessons from the old lane
An earlier CI lane for these tests (removed in #1159) had two defects this
PR specifically avoids:
- It ran pytest with `-k "sedonaspark"`, which matched zero tests and
silently ran nothing. The new run step has **no** `-k` selector.
- Its cache key hashed `raster_testing.py`, which does not contain the jar
pins, so the cache never invalidated when the pinned versions changed. The new
key hashes `testing_spark.py`, which does.
## Verification
The workflow triggers on changes to `.github/workflows/python.yml`, so this
PR's own CI run exercises the new lane end-to-end.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
