tustvold commented on issue #1473: URL: https://github.com/apache/arrow-rs/issues/1473#issuecomment-1102506402
Interesting, what hardware are you running on? I am very surprised to see `tokio_par_async_spawn_blocking_test` performing better than `tokio_sync_file_test`. For comparison some M1 benchmarks collected by @e-dard ``` sync_file_test (1024): min: 0.0746s, max: 0.0773s, avg: 0.0752s sync_mem_test (1024): min: 0.0717s, max: 0.0731s, avg: 0.0727s par_sync_file_test (1024): min: 0.0623s, max: 0.0898s, avg: 0.0634s tokio_sync_file_test (1024): min: 0.0626s, max: 0.0667s, avg: 0.0644s tokio_spawn_file_test (1024): min: 0.0899s, max: 0.1071s, avg: 0.0936s tokio_spawn_file_buffer_test (1024): min: 0.1009s, max: 0.1082s, avg: 0.1018s tokio_async_test (1024): min: 0.0915s, max: 0.0967s, avg: 0.0932s tokio_par_async_test (1024): min: 0.0841s, max: 0.0974s, avg: 0.0874s tokio_par_sync_test (1024): min: 0.0626s, max: 0.0835s, avg: 0.0668s sync_file_test (2048): min: 0.0653s, max: 0.1107s, avg: 0.0694s sync_mem_test (2048): min: 0.0622s, max: 0.0664s, avg: 0.0627s par_sync_file_test (2048): min: 0.0542s, max: 0.0568s, avg: 0.0551s tokio_sync_file_test (2048): min: 0.0547s, max: 0.0741s, avg: 0.0569s tokio_spawn_file_test (2048): min: 0.0724s, max: 0.0852s, avg: 0.0759s tokio_spawn_file_buffer_test (2048): min: 0.0907s, max: 0.1365s, avg: 0.0928s tokio_async_test (2048): min: 0.0814s, max: 0.0879s, avg: 0.0835s tokio_par_async_test (2048): min: 0.0761s, max: 0.0859s, avg: 0.0787s tokio_par_sync_test (2048): min: 0.0547s, max: 0.0968s, avg: 0.0584s sync_file_test (4096): min: 0.0605s, max: 0.0898s, avg: 0.0637s sync_mem_test (4096): min: 0.0579s, max: 0.0753s, avg: 0.0595s par_sync_file_test (4096): min: 0.0498s, max: 0.0596s, avg: 0.0543s tokio_sync_file_test (4096): min: 0.0498s, max: 0.0623s, avg: 0.0537s tokio_spawn_file_test (4096): min: 0.0639s, max: 0.0738s, avg: 0.0687s tokio_spawn_file_buffer_test (4096): min: 0.0856s, max: 0.1042s, avg: 0.0869s tokio_async_test (4096): min: 0.0759s, max: 0.0808s, avg: 0.0779s tokio_par_async_test (4096): min: 0.0714s, max: 0.0793s, avg: 0.0743s tokio_par_sync_test (4096): min: 0.0499s, max: 0.0610s, avg: 0.0552s sync_file_test (8192): min: 0.0576s, max: 0.0650s, avg: 0.0604s sync_mem_test (8192): min: 0.0544s, max: 0.0594s, avg: 0.0555s par_sync_file_test (8192): min: 0.0473s, max: 0.0571s, avg: 0.0514s tokio_sync_file_test (8192): min: 0.0475s, max: 0.0594s, avg: 0.0517s tokio_spawn_file_test (8192): min: 0.0587s, max: 0.0701s, avg: 0.0600s tokio_spawn_file_buffer_test (8192): min: 0.0821s, max: 0.1121s, avg: 0.0835s tokio_async_test (8192): min: 0.0728s, max: 0.0781s, avg: 0.0746s tokio_par_async_test (8192): min: 0.0686s, max: 0.0745s, avg: 0.0710s tokio_par_sync_test (8192): min: 0.0476s, max: 0.0568s, avg: 0.0514s sync_file_test (16384): min: 0.0566s, max: 0.0663s, avg: 0.0604s sync_mem_test (16384): min: 0.0543s, max: 0.0889s, avg: 0.0555s par_sync_file_test (16384): min: 0.0469s, max: 0.0603s, avg: 0.0509s tokio_sync_file_test (16384): min: 0.0471s, max: 0.0566s, avg: 0.0513s tokio_spawn_file_test (16384): min: 0.0624s, max: 0.0674s, avg: 0.0658s tokio_spawn_file_buffer_test (16384): min: 0.0871s, max: 0.0916s, avg: 0.0887s tokio_async_test (16384): min: 0.0726s, max: 0.0804s, avg: 0.0762s tokio_par_async_test (16384): min: 0.0683s, max: 0.0826s, avg: 0.0714s tokio_par_sync_test (16384): min: 0.0470s, max: 0.0567s, avg: 0.0510s ``` And an Intel Mac from @alamb ``` sync_file_test (1024): min: 0.0968s, max: 0.1207s, avg: 0.1027s sync_mem_test (1024): min: 0.0941s, max: 0.1220s, avg: 0.0996s par_sync_file_test (1024): min: 0.0830s, max: 0.1085s, avg: 0.0875s tokio_sync_file_test (1024): min: 0.0835s, max: 0.1080s, avg: 0.0917s tokio_spawn_file_test (1024): min: 0.1341s, max: 0.1747s, avg: 0.1460s tokio_spawn_file_buffer_test (1024): min: 0.1478s, max: 0.2100s, avg: 0.1600s tokio_async_test (1024): min: 0.1431s, max: 0.1804s, avg: 0.1526s tokio_par_async_test (1024): min: 0.1352s, max: 0.1612s, avg: 0.1465s tokio_par_sync_test (1024): min: 0.0880s, max: 0.1382s, avg: 0.1071s sync_file_test (2048): min: 0.0948s, max: 0.1768s, avg: 0.1318s sync_mem_test (2048): min: 0.0916s, max: 0.1666s, avg: 0.1206s par_sync_file_test (2048): min: 0.0794s, max: 0.1630s, avg: 0.1145s tokio_sync_file_test (2048): min: 0.1023s, max: 0.2510s, avg: 0.1902s tokio_spawn_file_test (2048): min: 0.2702s, max: 0.4230s, avg: 0.3572s tokio_spawn_file_buffer_test (2048): min: 0.2147s, max: 0.3401s, avg: 0.2680s tokio_async_test (2048): min: 0.2072s, max: 0.3323s, avg: 0.2596s tokio_par_async_test (2048): min: 0.1819s, max: 0.2990s, avg: 0.2395s tokio_par_sync_test (2048): min: 0.1382s, max: 0.2506s, avg: 0.1960s sync_file_test (4096): min: 0.1420s, max: 0.2399s, avg: 0.1813s sync_mem_test (4096): min: 0.1311s, max: 0.2399s, avg: 0.1782s par_sync_file_test (4096): min: 0.1079s, max: 0.1974s, avg: 0.1534s tokio_sync_file_test (4096): min: 0.1147s, max: 0.2118s, avg: 0.1638s tokio_spawn_file_test (4096): min: 0.2236s, max: 0.3454s, avg: 0.2782s tokio_spawn_file_buffer_test (4096): min: 0.1838s, max: 0.2941s, avg: 0.2340s tokio_async_test (4096): min: 0.1808s, max: 0.2900s, avg: 0.2255s tokio_par_async_test (4096): min: 0.1526s, max: 0.2556s, avg: 0.2048s tokio_par_sync_test (4096): min: 0.1141s, max: 0.2171s, avg: 0.1619s sync_file_test (8192): min: 0.1505s, max: 0.3689s, avg: 0.2261s sync_mem_test (8192): min: 0.1558s, max: 0.4034s, avg: 0.2375s par_sync_file_test (8192): min: 0.1149s, max: 0.2036s, avg: 0.1571s tokio_sync_file_test (8192): min: 0.1135s, max: 0.2022s, avg: 0.1614s tokio_spawn_file_test (8192): min: 0.3430s, max: 0.5689s, avg: 0.4529s tokio_spawn_file_buffer_test (8192): min: 0.1879s, max: 0.3810s, avg: 0.2620s tokio_async_test (8192): min: 0.1600s, max: 2.4110s, avg: 0.3616s tokio_par_async_test (8192): min: 0.1781s, max: 0.4306s, avg: 0.2548s tokio_par_sync_test (8192): min: 0.1223s, max: 0.3303s, avg: 0.2035s sync_file_test (16384): min: 0.1890s, max: 0.4439s, avg: 0.3216s sync_mem_test (16384): min: 0.1603s, max: 0.4605s, avg: 0.2934s par_sync_file_test (16384): min: 0.1184s, max: 0.3450s, avg: 0.2003s tokio_sync_file_test (16384): min: 0.1197s, max: 0.3192s, avg: 0.1841s tokio_spawn_file_test (16384): min: 0.2144s, max: 0.5118s, avg: 0.3662s tokio_spawn_file_buffer_test (16384): min: 0.2080s, max: 0.5335s, avg: 0.3274s tokio_async_test (16384): min: 0.1867s, max: 0.4569s, avg: 0.2526s tokio_par_async_test (16384): min: 0.1711s, max: 0.2684s, avg: 0.2118s tokio_par_sync_test (16384): min: 0.0868s, max: 0.2195s, avg: 0.1579s ``` Admittedly if memory serves these were collected before I made some further optimisations to the async reader, but I would be surprised -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
