comphead commented on issue #9359:
URL: https://github.com/apache/datafusion/issues/9359#issuecomment-2153456513
I made it work that way (notice `-m 1m`)
```
./target/debug/datafusion-cli -m 1m
DataFusion CLI v38.0.0
> set datafusion.optimizer.prefer_hash_join = false;
0 row(s) fetched.
Elapsed 0.011 seconds.
> set datafusion.execution.sort_spill_reservation_bytes = 104857;
0 row(s) fetched.
Elapsed 0.002 seconds.
> set datafusion.execution.sort_in_place_threshold_bytes = 104857;
0 row(s) fetched.
Elapsed 0.002 seconds.
> select * from (select unnest(range(0, 100000)) id) t inner join (select
unnest(range(0, 100000)) id) t1 on t.id = t1.id;
Resources exhausted: Failed to allocate additional 164552 bytes for
SMJStream[14] with 0 bytes already allocated - maximum available is 113800
>
```
Weird the same thing as a test doesn't respect memory pool
```
#[tokio::test]
async fn test_smj_spill() -> Result<()> {
let rt_config =
datafusion_execution::runtime_env::RuntimeConfig::new();
let rt_config = rt_config
.with_memory_pool(Arc::new(datafusion_execution::memory_pool::GreedyMemoryPool::new(1048576)));
let runtime_env = RuntimeEnv::new(rt_config);
let session_config =
SessionConfig::from_env()?.with_information_schema(true)
.with_sort_spill_reservation_bytes(104857)
.with_sort_in_place_threshold_bytes(104857);
let ctx = SessionContext::new_with_config_rt(session_config,
Arc::new(runtime_env?));
let sql = "set datafusion.optimizer.prefer_hash_join = false;";
let _ = ctx.sql(sql).await?.collect().await?;
let sql = "
select * from (select unnest(range(0, 100000)) id) t inner join (select
unnest(range(0, 100000)) id) t1 on t.id = t1.id
";
let actual = ctx.sql(sql).await?.collect().await?;
println!("{}", actual.len());
println!("=============================================================================");
Ok(())
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
