comphead commented on issue #19815: URL: https://github.com/apache/datafusion/issues/19815#issuecomment-6744782547
### DataFusion vs Polars, TPC-H SF1000 This follows up on the DataFusion SF1000 numbers above. Polars ran on the same hardware against the same data. **Setup** - DataFusion `57.0.0-dev`, `main` at [`a616831`](https://github.com/apache/datafusion/commit/a616831122aa525ed843ee3c5e11f4803f557c98), `target_partitions = 32`, `memory_limit = 105G` (greedy pool) - Polars `2.1.0` at [`7f902bd`](https://github.com/pola-rs/polars/commit/7f902bd3ffd27ff82d14278a09a295ff2f7aed92), streaming engine, `POLARS_TEMP_DIR=/data`, no explicit memory limit - Hardware: k8s pod, amd64, 32 cores - Data: TPC-H SF1000 Parquet, zstd, 512 MB blocks, partitioned, read from S3 - 3 iterations per query. The times below are mean seconds. **Results** | Query | DataFusion | Polars | DF / Polars | |---|---:|---:|---:| | Q01 | 46.17 | 26.53 | 1.74x | | Q02 | 37.71 | 6.96 | 5.42x | | Q03 | 31.25 | 23.54 | 1.33x | | Q04 | 22.92 | 41.69 | 0.55x | | Q05 | 53.85 | 34.42 | 1.56x | | Q06 | 4.76 | 4.71 | 1.01x | | Q07 | 61.13 | 38.12 | 1.60x | | Q08 | 54.80 | 40.72 | 1.35x | | Q09 | OOM | 73.52 | | | Q10 | 37.20 | deadlock | | | Q11 | 13.44 | 9.21 | 1.46x | | Q12 | 14.86 | 24.57 | 0.60x | | Q13 | 29.77 | 39.82 | 0.75x | | Q14 | 4.01 | 3.23 | 1.24x | | Q15 | 16.55 | 3.05 | 5.43x | | Q16 | 11.52 | 9.18 | 1.25x | | Q17 | 75.21 | 59.78 | 1.26x | | Q18 | OOM | deadlock | | | Q19 | 31.41 | 28.47 | 1.10x | | Q20 | 24.35 | 10.06 | 2.42x | | Q21 | OOM | deadlock | | | Q22 | 22.63 | 18.68 | 1.21x | Both engines finished 18 queries. Over those 18, DataFusion took 556.4 s and Polars took 422.7 s (1.32x). The geometric means are 23.9 s and 16.8 s (1.42x). ```mermaid %%{init: {"themeVariables": {"xyChart": {"plotColorPalette": "#2a78d6, #8a8984"}}}}%% xychart-beta title "DataFusion time / Polars time, SF1000 (above 1 = Polars faster)" x-axis ["Q01", "Q02", "Q03", "Q04", "Q05", "Q06", "Q07", "Q08", "Q11", "Q12", "Q13", "Q14", "Q15", "Q16", "Q17", "Q19", "Q20", "Q22"] y-axis "DF / Polars" 0 --> 6 bar [1.74, 5.42, 1.33, 0.55, 1.56, 1.01, 1.60, 1.35, 1.46, 0.60, 0.75, 1.24, 5.43, 1.25, 1.26, 1.10, 2.42, 1.21] line [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] ``` Observations: - DataFusion is faster on Q04, Q12 and Q13. It is about even on Q06. - The largest gaps are Q02 and Q15 (about 5.4x) and Q20 (2.4x). - DataFusion failed Q09, Q18 and Q21 with hash join build OOMs at 105G, as described above. Polars completed Q09 in 73.5 s. - Polars deadlocked on Q10, Q18 and Q21 and reported no result for them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
