shyjsarah opened a new issue, #673: URL: https://github.com/apache/paimon-rust/issues/673
### Motivation `paimon_datafusion::SQLContext` currently creates a DataFusion session with the default `RuntimeEnv`. Applications embedding Paimon Rust therefore cannot provide a bounded execution memory pool, configure a spill-aware pool, select a temporary directory, or cap temporary disk usage. Long-running services and other resource-constrained applications need to control these DataFusion runtime resources without replacing Paimon's SQL context and its custom planners, optimizer rules, catalogs, and functions. ### Proposal - Add a backward-compatible builder for `SQLContext`. - Allow Rust callers to provide a custom DataFusion `RuntimeEnv` while retaining all Paimon-specific session configuration. - Expose optional, generic memory-pool and temporary-directory settings from the Python `SQLContext` binding. - Keep `SQLContext::new()` and zero-argument Python `SQLContext()` behavior unchanged. ### Non-goals - Detecting container or host resource limits. - Managing application-specific temporary-directory lifecycle. - Guaranteeing spill support for operators that DataFusion itself cannot spill. - Accounting for allocations outside DataFusion's memory pool. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
