comphead commented on issue #19815: URL: https://github.com/apache/datafusion/issues/19815#issuecomment-6695803863
### DataFusion TPC-H SF1000 on `main` (`a616831`) **Setup** - DataFusion `57.0.0-dev`, `main` at [`a616831`](https://github.com/apache/datafusion/commit/a616831122aa525ed843ee3c5e11f4803f557c98) (2026-11-06) - Hardware: k8s pod, amd64, 32 cores - Data: TPC-H SF1000 Parquet, zstd, 512 MB blocks, partitioned, read from S3 - `target_partitions = 32` - `datafusion.runtime.memory_limit = 105G` (greedy pool) - `datafusion.runtime.temp_directory = /data`, `max_temp_directory_size = 900G` - `datafusion.runtime.metadata_cache_limit = 2G`, `list_files_cache_limit = 256M`, `file_statistics_cache_limit = 256M` - 3 iterations per query. The times below are in seconds. This commit already includes the revert of #25790 (#27145), so the Q18 failure below is not the #27108 regression. **Results** | Query | Mean | Min | Max | Status | |---|---:|---:|---:|---| | Q01 | 46.17 | 45.86 | 46.38 | ok | | Q02 | 37.71 | 35.83 | 41.20 | ok | | Q03 | 31.25 | 29.53 | 34.22 | ok | | Q04 | 22.92 | 22.58 | 23.18 | ok | | Q05 | 53.85 | 47.61 | 64.85 | ok | | Q06 | 4.76 | 4.63 | 4.84 | ok | | Q07 | 61.13 | 59.27 | 62.28 | ok | | Q08 | 54.80 | 53.02 | 56.15 | ok | | Q09 | | | | **failed (OOM)** | | Q10 | 37.20 | 36.27 | 38.02 | ok | | Q11 | 13.44 | 12.13 | 14.64 | ok | | Q12 | 14.86 | 13.94 | 15.80 | ok | | Q13 | 29.77 | 29.43 | 30.32 | ok | | Q14 | 4.01 | 3.50 | 4.57 | ok | | Q15 | 16.55 | 10.93 | 22.36 | ok | | Q16 | 11.52 | 10.22 | 13.70 | ok | | Q17 | 75.21 | 74.55 | 76.05 | ok | | Q18 | | | | **failed (OOM)** | | Q19 | 31.41 | 30.92 | 31.77 | ok | | Q20 | 24.35 | 23.23 | 25.99 | ok | | Q21 | | | | **failed (OOM)** | | Q22 | 22.63 | 20.17 | 24.50 | ok | 19 of 22 queries passed. The sum of their means is 593.6 s and the geometric mean is 24.4 s. **Failures** Q09, Q18 and Q21 failed on all 3 iterations with `Resources exhausted` while building hash join input. `HashJoinInput` reports `can spill: false`, so the 105G pool fills up with build-side reservations of similar size, one per partition. - **Q09**: `Failed to allocate additional 763.0 MB for HashJoinInput[24] with 763.0 MB already allocated for this reservation - 361.1 MB remain available for the total memory pool: greedy(used: 104.6 GB, pool_size: 105.0 GB)`. The top consumers are `HashJoinInput` partitions at 2.1 GB each. - **Q18**: `Failed to allocate additional 1440.0 KB for HashJoinInput[4] with 2.5 GB already allocated for this reservation - 247.2 KB remain available for the total memory pool: greedy(used: 105.0 GB, pool_size: 105.0 GB)`. The top consumers are `HashJoinInput` partitions at 2.5 GB each. - **Q21**: `Failed to allocate additional 4.7 GB for HashJoinInput[24] with 0.0 B already allocated for this reservation - 1591.6 MB remain available for the total memory pool: greedy(used: 103.4 GB, pool_size: 105.0 GB)`. The top consumers are `HashJoinInput` partitions at 6.5 GB each. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
