Smotrov commented on issue #10897:
URL: https://github.com/apache/datafusion/issues/10897#issuecomment-2173801556
Thank you @alamb
This is what I actually did.
```RUST
const MEMORY_LIMIT: usize = 8 * 1024 * 1024 * 1024; // 8GB
fn create_context() -> Result<SessionContext> {
// Create a memory pool with a limit
let memory_pool = Arc::new(FairSpillPool::new(MEMORY_LIMIT));
// Configure the runtime environment to use the memory pool
let rt_config = RuntimeConfig::new()
.with_memory_pool(memory_pool)
.with_temp_file_path(PathBuf::from("./tmp"));
let runtime_env = Arc::new(RuntimeEnv::new(rt_config)?);
// Configure the session context to use the runtime environment
let session_config = SessionConfig::new();
let ctx = SessionContext::new_with_config_rt(session_config,
runtime_env);
Ok(ctx)
}
```
However it easily takes 20..30GB of RAM and what is interesting, the CPU
load stays relatively small. Like 20...30%.
The memory consumption is that high when I set at lease 4 target partitions.
```RUST
// Define the partitioned Listing Table
let listing_options = ListingOptions::new(file_format)
.with_table_partition_cols(part)
.with_target_partitions(4)
.with_file_extension(".ndjson.zst");
```
Would be grate if it would be possible to set an actual limit for the memory
otherwise I can't use it on docker :-(
Full utilization of all CPU cores would also be cool if possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
