alamb opened a new issue, #20524:
URL: https://github.com/apache/datafusion/issues/20524
### Is your feature request related to a problem or challenge?
@blaginin is trying to speed up execution of tests
Right now, when running the tests by default it takes 8 wallclock seconds
with 16 cores.
```shell
cargo test --profile=ci --test sqllogictests
...
Running with 16 test threads (available parallelism: 16)
Completed 407 test files in 8 seconds
```
However, if you run `time`, you can see this only consumes 53 core*sec out
of a theoretical 128 core*sec ( 16 cores * 8 seconds = 128 core*sec)
```shell
time cargo test --profile=ci --test sqllogictests
Finished `ci` profile [unoptimized] target(s) in 0.34s
Running bin/sqllogictests.rs
(target/ci/deps/sqllogictests-91ecf623e0e3b627)
Running with 16 test threads (available parallelism: 16)
Completed 407 test files in 8 seconds
real 0m8.232s
user 0m53.201s
sys 0m1.247s
```
This means the tests could in theory go twice as fast with more cores
I when looked briefly at this, I think one problem is that the
push_down_filter.slt test takes a long time and is single threaded for that time
```
[00:00:05] ####################-------------------- 52/107
"push_down_filter.slt" - 4 took > 500 ms
```
### Describe the solution you'd like
I would like the sqllogictests to run on all cores
### Describe alternatives you've considered
Maybe we can start by splitting up push_down_filter.slt into a few smaller
files that can run in parallel
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
