jiayuasu opened a new pull request, #1155: URL: https://github.com/apache/sedona-db/pull/1155
## What changes are proposed in this pull request? The 0.4.0 release enabled memory-limited execution by default: `SedonaContextBuilder::new()` set the memory limit to 75% of physical memory with a fair spill pool. Under this default, spilling operators can write large amounts of intermediate data to disk, which broke SpatialBench SF1 Q12 and SF10 Q5/Q12 with `No space left on device` errors and slowed down queries that previously ran fine. This PR reverts the default to unlimited memory so spilling is opt-in: - `SedonaContextBuilder::new()` no longer sets a memory limit by default; DataFusion's unbounded memory pool is used unless a limit is explicitly configured via the builder, the CLI `--memory-limit` flag, or context options. Setting a limit still enables the fair pool and spill-to-disk behavior exactly as before. - Removed the now-unused `default_memory_limit()` helper and the `sysinfo` dependency. - Updated the CLI help text, Python option docstrings, and the memory-management guide (notebook + generated markdown) to describe the opt-in behavior. ## How was this change tested? - `cargo test -p sedona --lib context_builder` (19 tests, updated to assert the new default) - `cargo test -p sedona --lib test_auto_configure_spilled_batch_threshold` - `cargo test -p sedona --doc context_builder` - `cargo check -p sedona-cli` - pre-commit on all changed files ## Does this PR introduce any user-facing change? Yes. Contexts no longer enforce a memory limit by default; memory-limited execution with spill-to-disk is now opt-in via `memory_limit`. This restores the pre-0.4.0 default behavior. Closes #1040 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
