kornholi commented on pull request #798: URL: https://github.com/apache/arrow-rs/pull/798#issuecomment-930601692
Sorry, the encoding remark is slightly misleading as the `ByteArray` encoders buffer internally for a while. It's consistently 6-7% faster writing overall for the datasets I tested (e.g. ~116.9s to ~109.5s for 75M rows). @alamb - There's little the compiler can do here as comparing `val` to itself must always load the whole value, while the min/max comparisons can usually exit early. This PR shaves off at least 10% of overall memory loads in my case, but that includes a lot of unrelated processing. It depends on your input data of course! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
