jiayuasu opened a new pull request, #128: URL: https://github.com/apache/sedona-spatialbench/pull/128
## Summary Second of two PRs for #126. PR #127 committed the ground-truth answers; this PR adds the **correctness harness** that verifies every participating engine returns those answers, wired into CI. ## What's here **`benchmark/verify_correctness.py`** — for each `(engine, query)` it runs the query, normalizes the result to the same engine-neutral form the answers use, and compares against `benchmark/answers/sf<sf>/q<n>.parquet`: - **PASS** — matches within float tolerance (`rtol=1e-6`, `atol=1e-9`); exact for integer keys, strings, and timestamps. - **SKIP** — the engine can't compute the query in time (e.g. DuckDB's lateral-join Q12 at SF1) or no answer is committed. Skips don't fail the run. - **FAIL** — a real discrepancy; non-zero exit. Comparison details: - Columns are compared **by position**, since engines name the same column differently (e.g. `avg_duration` vs `avg_duration_seconds`). - A within-tolerance difference confined to the final **`LIMIT`-boundary row** passes (float ties can order that row differently across engines). **`.github/workflows/correctness.yml`** — runs the harness per engine at **SF1** on PRs touching `benchmark/**`, `spatialbench-queries/**`, or the workflow, gating on any mismatch. ## Engine isolation (learned building the answers generator) Each `(engine, query)` runs in its own subprocess: - **sedonadb** is imported before pyarrow (else `.to_pandas()` segfaults), and the worker calls `os._exit(0)` to skip sedonadb 0.4.0's teardown segfault; - the worker pickles its result (never touching pyarrow's parquet writer, which conflicts with sedonadb's bundled Arrow); - a per-query timeout turns a hanging engine into a SKIP rather than a stuck job; - the **pure-pandas parent** reads the answer parquet and does the comparison. ## Local validation Ran at SF1 against the committed answers: - **SedonaDB** (the oracle): 12/12 PASS. - **DuckDB**: PASS on the queries it can compute; Q12 SKIP (lateral join times out). - **GeoPandas**: PASS on the sampled queries. - Deliberately corrupting an answer value produces a FAIL with a precise per-column/row message and a non-zero exit. Spatial Polars and PyCanopy use the identical worker path and are exercised by CI. ## Notes / follow-ups - Answers exist for **SF1** only, so verification runs at SF1; **SF10** answers + verification to follow. - The generator that produced the answers is intentionally not committed (per review on #127); the harness is self-contained (it only reads the committed parquet answers). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
