haohuaijin commented on PR #9536:
URL:
https://github.com/apache/arrow-datafusion/pull/9536#issuecomment-1991126078
@simonvandel and @alamb, I chose 1000 instead of 700 because I noticed that
the plan time does not increase linearly with the number of columns. This is
evident from the benchmarks shown below. I separately ran benchmarks for select
* from t700, t1000, and t1500. The plan time increased from 289ms for t700 to
1.3s for t1500. This is a 5x slowdown, even though the number of columns only
doubled. But I don't have a strong opinion on wanting to keep 1000, 700 also
mean selecting many columns in a huge table. Do I need to change it?
```
logical_select_all_from_700
time: [39.054 ms 39.098 ms 39.144 ms]
Found 4 outliers among 100 measurements (4.00%)
3 (3.00%) high mild
1 (1.00%) high severe
Benchmarking physical_select_all_from_700: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase
target time to 28.9s, or reduce sample count to 10.
physical_select_all_from_700
time: [287.53 ms 289.03 ms 291.39 ms]
Found 3 outliers among 100 measurements (3.00%)
3 (3.00%) high severe
Benchmarking logical_select_all_from_1000: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase
target time to 8.5s, or reduce sample count to 50.
logical_select_all_from_1000
time: [85.450 ms 85.689 ms 85.945 ms]
Found 4 outliers among 100 measurements (4.00%)
4 (4.00%) high mild
Benchmarking physical_select_all_from_1000: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase
target time to 62.7s, or reduce sample count to 10.
physical_select_all_from_1000
time: [626.48 ms 627.34 ms 628.22 ms]
Found 3 outliers among 100 measurements (3.00%)
3 (3.00%) high mild
Benchmarking logical_select_all_from_1500: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase
target time to 17.9s, or reduce sample count to 20.
logical_select_all_from_1500
time: [179.09 ms 179.34 ms 179.59 ms]
Benchmarking physical_select_all_from_1500: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase
target time to 132.7s, or reduce sample count to 10.
physical_select_all_from_1500
time: [1.3271 s 1.3289 s 1.3308 s]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
