yordan-pavlov commented on issue #200: URL: https://github.com/apache/arrow-rs/issues/200#issuecomment-835459167
UPDATE: after some more benchmarking and some tweaks, the new arrow array reader is now consistently faster in all cases, when reading string arrays, with performance improvement between 14% and 44%. As a next step I would like to also benchmark performance of reading primitive arrays, for a more complete picture, before I create a PR. Here are the latest benchmark results: read StringArray, plain encoded, mandatory, no NULLs - old: time: [1.6621 ms 1.6881 ms 1.7159 ms] read StringArray, plain encoded, mandatory, no NULLs - new: time: [933.62 us 944.74 us 959.72 us] read StringArray, plain encoded, optional, no NULLs - old: time: [1.7153 ms 1.7293 ms 1.7462 ms] read StringArray, plain encoded, optional, no NULLs - new: time: [1.0153 ms 1.0186 ms 1.0221 ms] read StringArray, plain encoded, optional, half NULLs - old: time: [1.4809 ms 1.5016 ms 1.5241 ms] read StringArray, plain encoded, optional, half NULLs - new: time: [889.95 us 903.29 us 919.65 us] read StringArray, dictionary encoded, mandatory, no NULLs - old: time: [1.4234 ms 1.4501 ms 1.4812 ms] read StringArray, dictionary encoded, mandatory, no NULLs - new: time: [1.2338 ms 1.2401 ms 1.2473 ms] read StringArray, dictionary encoded, optional, no NULLs - old: time: [1.5339 ms 1.5489 ms 1.5662 ms] read StringArray, dictionary encoded, optional, no NULLs - new: time: [1.2835 ms 1.2913 ms 1.3010 ms] read StringArray, dictionary encoded, optional, half NULLs - old: time: [1.3033 ms 1.3308 ms 1.3640 ms] read StringArray, dictionary encoded, optional, half NULLs - new: time: [984.41 us 992.85 us 1.0026 ms] and here are the latest changes: https://github.com/yordan-pavlov/arrow/commit/48138d65355ccd520ee7bb8be5d7db31f026d550 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
