isidentical commented on PR #4128: URL: https://github.com/apache/arrow-datafusion/pull/4128#issuecomment-1305943969
While playing with this, I've also written a little Python script to function like a benchmark comparison UI (poor man's conbench): https://gist.github.com/isidentical/4e3fff1350e9d49672e15d54d9e8299f It is quite basic, but I think it can automate a few stuff for https://github.com/datafusion-contrib/benchmark-automation/tree/main. E.g. an example comparison between ``` $ ./target/release/tpch benchmark datafusion --path /opt/data-parquet --format parquet --iterations 3 -o /tmp/benchmarks --disable-statistics $ ./target/release/tpch benchmark datafusion --path /opt/data-parquet --format parquet --iterations 3 -o /tmp/benchmarks $ python t.py compare /tmp/benchmarks/file1.json /tmp/benchmarks/file2.json ┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Query ┃ Baseline ┃ Comparison ┃ Change ┃ ┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ Q1 │ 702.18ms │ 687.86ms │ no change │ │ Q2 │ 413.74ms │ 302.22ms │ +1.37x faster │ │ Q3 │ 392.94ms │ 395.34ms │ no change │ │ Q4 │ 111.28ms │ 97.01ms │ +1.15x faster │ │ Q5 │ 465.81ms │ 487.92ms │ no change │ │ Q6 │ 402.94ms │ 402.48ms │ no change │ │ Q7 │ 868.18ms │ 889.51ms │ no change │ │ Q8 │ 499.98ms │ 468.68ms │ +1.07x faster │ │ Q9 │ 827.54ms │ 837.67ms │ no change │ │ Q10 │ 503.22ms │ 492.29ms │ no change │ │ Q11 │ 221.30ms │ 167.37ms │ +1.32x faster │ │ Q12 │ 204.10ms │ 170.99ms │ +1.19x faster │ │ Q13 │ 441.50ms │ 423.67ms │ no change │ │ Q14 │ 373.42ms │ 383.57ms │ no change │ │ Q15 │ 356.24ms │ 352.67ms │ no change │ │ Q16 │ 115.38ms │ 117.98ms │ no change │ │ Q17 │ 2099.22ms │ 2209.00ms │ 1.05x slower │ │ Q18 │ 1255.95ms │ 1285.39ms │ no change │ │ Q19 │ 656.93ms │ 660.46ms │ no change │ │ Q20 │ 640.30ms │ 624.94ms │ no change │ │ Q21 │ 697.55ms │ 685.22ms │ no change │ │ Q22 │ 84.20ms │ 81.76ms │ no change │ └──────────────┴──────────────┴──────────────┴───────────────┘ ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
