yordan-pavlov edited a comment on issue #200: URL: https://github.com/apache/arrow-rs/issues/200#issuecomment-831571253
I wrote some benchmarks over the weekend, and the performance improvement ranges from about 10% to about 40% depending on the type of encoding (plain vs dictionary) and type of column (optional vs mandatory), except for the 50% NULL values case where the new implementation is slower. I will be looking at the code again later this week, to see if I can make some adjustments to improve performance further. Here are the results: **plain-encoded string values** read StringArray, plain encoded, mandatory, no NULLs - old: time: [3.2249 ms 3.2446 ms 3.2677 ms] read StringArray, plain encoded, mandatory, no NULLs - new: time: [1.9518 ms 1.9721 ms 1.9952 ms] read StringArray, plain encoded, optional, no NULLs - old: time: [3.5536 ms 3.5909 ms 3.6302 ms] read StringArray, plain encoded, optional, no NULLs - new: time: [2.2593 ms 2.2807 ms 2.3042 ms] read StringArray, plain encoded, optional, half NULLs - old: time: [3.0227 ms 3.0588 ms 3.0978 ms] read StringArray, plain encoded, optional, half NULLs - new: time: [3.2593 ms 3.3919 ms 3.5220 ms] **dictionary-encoded string values** read StringArray, dictionary encoded, mandatory, no NULLs - old: time: [2.9892 ms 3.0591 ms 3.1435 ms] read StringArray, dictionary encoded, mandatory, no NULLs - new: time: [2.7054 ms 2.7271 ms 2.7497 ms] read StringArray, dictionary encoded, optional, no NULLs - old: time: [3.1137 ms 3.1453 ms 3.1809 ms] read StringArray, dictionary encoded, optional, no NULLs - new: time: [2.9156 ms 2.9581 ms 3.0054 ms] read StringArray, dictionary encoded, optional, half NULLs - old: time: [2.7450 ms 2.7824 ms 2.8196 ms] read StringArray, dictionary encoded, optional, half NULLs - new: time: [3.4127 ms 3.4430 ms 3.4747 ms] -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
