Dandandan commented on issue #20324: URL: https://github.com/apache/datafusion/issues/20324#issuecomment-3902431429
I did some PoC with adding an option to skip filter pushdown in the reader based on the selectivity: https://github.com/apache/arrow-rs/pull/9414 This seems to correlate with some of the reported slowdowns. ``` arrow_reader_clickbench/async_object_store/Q19 time: [2.3314 ms 2.3414 ms 2.3545 ms] change: [−10.565% −8.7295% −7.0407%] (p = 0.00 < 0.05) Performance has improved. Found 3 outliers among 100 measurements (3.00%) arrow_reader_clickbench/async_object_store/Q36 time: [14.871 ms 14.899 ms 14.929 ms] change: [−15.341% −14.945% −14.564%] (p = 0.00 < 0.05) Performance has improved. Found 5 outliers among 100 measurements (5.00%) 4 (4.00%) high mild 1 (1.00%) high severe arrow_reader_clickbench/async_object_store/Q37 time: [4.3473 ms 4.4337 ms 4.5527 ms] change: [−29.402% −27.837% −25.858%] (p = 0.00 < 0.05) Performance has improved. Found 6 outliers among 100 measurements (6.00%) 3 (3.00%) high mild 3 (3.00%) high severe arrow_reader_clickbench/async_object_store/Q38 time: [13.772 ms 13.800 ms 13.828 ms] change: [−6.0804% −5.4308% −4.8014%] (p = 0.00 < 0.05) Performance has improved. Found 2 outliers among 100 measurements (2.00%) 1 (1.00%) low mild 1 (1.00%) high mild arrow_reader_clickbench/async_object_store/Q39 time: [25.469 ms 25.564 ms 25.688 ms] change: [−0.7996% −0.1285% +0.5766%] (p = 0.71 > 0.05) No change in performance detected. Found 4 outliers among 100 measurements (4.00%) 3 (3.00%) high mild 1 (1.00%) high severe arrow_reader_clickbench/async_object_store/Q40 time: [4.6984 ms 4.7729 ms 4.8755 ms] change: [−29.897% −27.901% −25.779%] (p = 0.00 < 0.05) Performance has improved. Found 13 outliers among 100 measurements (13.00%) 3 (3.00%) high mild 10 (10.00%) high severe arrow_reader_clickbench/async_object_store/Q41 time: [4.0334 ms 4.0579 ms 4.0897 ms] change: [−23.479% −22.838% −22.089%] (p = 0.00 < 0.05) Performance has improved. Found 8 outliers among 100 measurements (8.00%) 5 (5.00%) high mild 3 (3.00%) high severe arrow_reader_clickbench/async_object_store/Q42 time: [3.1349 ms 3.1465 ms 3.1598 ms] change: [−9.4234% −7.3366% −5.8732%] (p = 0.00 < 0.05) Performance has improved. Found 9 outliers among 100 measurements (9.00%) 5 (5.00%) high mild 4 (4.00%) high severe ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
