shyjsarah opened a new pull request, #675:
URL: https://github.com/apache/paimon-rust/pull/675
### Purpose
Linked issue: close #673
`SQLContext` currently always creates a DataFusion session with the default
`RuntimeEnv`. Applications embedding Paimon Rust cannot provide a bounded
execution memory pool, choose a spill policy, select a temporary directory, or
cap temporary disk usage without replacing Paimon's SQL context and its custom
planners and functions.
This change makes DataFusion runtime resources configurable while preserving
the existing zero-configuration behavior.
### Brief change log
- Add a public, backward-compatible `SQLContextBuilder` that accepts an
`Arc<RuntimeEnv>`.
- Keep `SQLContext::new()` unchanged and delegate its construction through
the builder.
- Add optional keyword-only Python `SQLContext` arguments for:
- `memory_pool_type` (`fair` or `greedy`)
- `memory_pool_bytes`
- `temp_directory`
- `max_temp_directory_size_bytes`
- Update Python type stubs and user documentation.
- Add Rust coverage for custom `RuntimeEnv` injection.
- Add a Python regression test that executes an external sort and verifies
that DataFusion reports spilled rows and bytes.
### Tests
- `cargo fmt --all -- --check`
- `PYO3_PYTHON=bindings/python/.venv/bin/python cargo clippy -p
paimon-datafusion -p pypaimon_rust --all-targets --features fulltext -- -D
warnings`
- `cargo test -p paimon-datafusion
test_sql_context_builder_uses_custom_runtime_env --lib`
- `make build`
- `uv run --no-sync pytest tests/test_datafusion.py -k
'runtime_resource_configuration' -q`
### API and Format
This adds public Rust and Python APIs. Existing constructors remain backward
compatible. There is no storage-format change.
### Documentation
The Python binding documentation now includes a bounded-memory and
local-spill example and clarifies that DataFusion's memory pool does not
account for every host or external-library allocation.
### AI assistance
AI tooling was used to help implement and test this change. I reviewed the
builder, Python binding, DataFusion memory-pool selection, temporary-directory
behavior, and compatibility path end-to-end. The primary known limitation is
the one documented above: DataFusion memory pools only account for allocations
registered with the pool, and only spill-capable operators can move
intermediate state to disk.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
