tustvold commented on pull request #1284: URL: https://github.com/apache/arrow-rs/pull/1284#issuecomment-1040604407
At least for IOx, we're in control of the parquet data written and so it is a case of us choosing to write using the encodings that give us the best balance of compression and performance. Ultimately I saw this decoder show up in a profile, as the dictionary encoding appears to spill to this, and realised there was clearly some low-hanging fruit here and so coded it up. I'm personally optimistic there are further potential improvements that will make DeltaBinaryPacked have comparable decode performance to PLAIN, at which point it effectively becomes free compression, but I don't know that for sure :sweat_smile: So to directly answer your question, we do have production use cases of DeltaBinaryPacked encoding, but this is somewhat accidental and we would likely switch to something else should it yield better performance characteristics. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
