yjshen commented on PR #2146:
URL:
https://github.com/apache/arrow-datafusion/pull/2146#issuecomment-1087143111
# TPC-H SF=10
`master`
```
target/release/tpch benchmark datafusion --iterations 3 --path
/home/yijie/sort_test/tpch-parquet-sf10 --format parquet --query 1 --batch-size
4096
Running benchmarks with the following options: DataFusionBenchmarkOpt {
query: 1, debug: false, iterations: 3, partitions: 2, batch_size: 4096, path:
"/home/yijie/sort_test/tpch-parquet-sf10", file_format: "parquet", mem_table:
false, output_path: None }
Query 1 iteration 0 took 47772.6 ms and returned 59986051 rows
Query 1 iteration 1 took 47899.2 ms and returned 59986051 rows
Query 1 iteration 2 took 48861.9 ms and returned 59986051 rows
Query 1 avg time: 48177.89 ms
```
This PR:
```
Running benchmarks with the following options: DataFusionBenchmarkOpt {
query: 1, debug: false, iterations: 3, partitions: 2, batch_size: 4096, path:
"/home/yijie/sort_test/tpch-parquet-sf10", file_format: "parquet", mem_table:
false, output_path: None }
Query 1 iteration 0 took 38565.1 ms and returned 59986051 rows
Query 1 iteration 1 took 37786.0 ms and returned 59986051 rows
Query 1 iteration 2 took 37056.7 ms and returned 59986051 rows
Query 1 avg time: 37802.62 ms
```
The performance has **improved** by ~21.5% this time. The advantage of
better memory accessing pattern pays off the extra computation for row <->
columnar transformation.
`master`
```
Performance counter stats for 'system wide':
9,443,323,018 cache-misses # 41.338 % of all cache
refs
22,844,399,240 cache-references
14,787,052,560 l3_cache_accesses
5,753,820,101 l3_misses
3,046,705,364 dTLB-load-misses # 54.75% of all dTLB
cache accesses
5,565,251,257 dTLB-loads
147.045336524 seconds time elapsed
```
This PR:
```
Performance counter stats for 'system wide':
6,750,648,518 cache-misses # 30.344 % of all cache
refs
22,247,021,905 cache-references
10,821,629,799 l3_cache_accesses
2,122,684,404 l3_misses
2,348,824,410 dTLB-load-misses # 64.09% of all dTLB
cache accesses
3,664,743,134 dTLB-loads
115.306819499 seconds time elapsed
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
