jiayuasu opened a new pull request, #1187: URL: https://github.com/apache/sedona-db/pull/1187
## What - add an `ExternalFormatSpec` capability for formats that cannot safely consume multiple file readers concurrently - disable concurrent file reads for pyogrio and serialize schema inference and the complete reader lifecycle within each physical scan - drop the Arrow FFI stream before closing its owning Python context and releasing the scan permit - add regression tests for scan-local serialization and deterministic Python-context cleanup ## Why #1051 serialized pyogrio context entry and exit, but active Arrow streams could still overlap. Subsequent wheel runs continued to abort during multi-file and partitioned pyogrio reads, as tracked in #1072. An unchanged CPython 3.12 comparison repeatedly ran the two affected tests in fresh processes and reproduced SIGABRT at iteration 72 during `test_read_ogr_partitioned`: [run 32625781579](https://github.com/jiayuasu/sedona-db/actions/runs/32625781579/job/97160805281). This change keeps one scan-local permit from reader creation through consumption and native cleanup. Pyogrio multi-file scans therefore trade file-level parallelism for stability. Other external formats retain concurrent reads by default, and independent physical scans do not share a process-wide lock. ## Verification - `cargo fmt --all -- --check` - `cargo test -p sedona-datasource -p sedonadb --lib` (9 + 11 tests passed) - `cargo clippy -p sedona-datasource -p sedonadb --lib --no-deps -- -D warnings` - local pyogrio and external-read modules (22 tests passed) - pinned CPython 3.12 stress run: - full pyogrio module: 18/18 passed - 500 same-process repetitions and 500 fresh-process repetitions - 2,002 executions of the two affected tests with no abort, hang, timeout, or test failure - [run 32628586406](https://github.com/jiayuasu/sedona-db/actions/runs/32628586406) - all five core wheel platforms passed on the combined verification branch: [run 32627022432](https://github.com/jiayuasu/sedona-db/actions/runs/32627022432) Closes #1072. Follow-up to #924 and #1051. Replacing the Python reader with the native GDAL reader remains tracked in #1092. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
