Hi Andrei, I tried finding more details on block compression in parquet (or compression per data page) and I couldn't find anything to satisfy my curiosity about how it can be used and how it performs. I hate being the person to just say "test it first," so I want to also recommend figuring out how you'd imagine the interface to be designed. Some formats like ORC seem to have 2 compression modes (optimize for speed or space) while parquet exposes more of the tuning knobs (according to [1]). And to Gang's point, there's a question of what can be exposed to the various abstraction levels (perhaps end users would never be interested in this so it's exposed only through an advanced or internal interface). Anyways, good luck scoping it out and feel free to iterate with the mailing list as you try things out rather than just when finished, maybe someone can chime in with more information and thoughts in the meantime. [1]: https://arxiv.org/pdf/2304.05028.pdf Sent from Proton Mail for iOS On Sat, Mar 23, 2024 at 05:23, Andrei Lazăr <lazarandrei...@gmail.com> wrote: Hi Aldrin, thanks for taking the time to reply to my email!
In my understanding, compression on Parquet files happens on the Data Page level for every column, meaning that even across a row group, there can be multiple units of data compression, and most certainly there are going to be different units of data compression across an entire Parquet file. Therefore, what I am hoping for is that more granular compression algorithm choices could lead to overall better compression as the data in the same column across row groups can differ quite a lot. At this very moment, specifying different compression algorithms per column is supported and in my use case it is extremely helpful, as I have some columns (mostly containing floats), for which a compression algorithm like Snappy (or even no compression at all), significantly speeds up my queries than keeping the data compressed with something like ZSTD or GZIP. That being said, your suggestion of writing a benchmark and sharing the results here to support considering this approach is a great idea, I will try doing that! Once again, thank you for your time! Kind regards, Andrei On Fri, 22 Mar 2024 at 22:12, Aldrin <octalene....@pm.me.invalid> wrote: > Hello! > > I don't do much with compression, so I could be wrong, but I assume a > compression algorithm spans the whole column and areas of large variance > generally benefit less from the compression, but the encoding still > provides benefits across separate areas (e.g. separate row groups). > > My impression is that compression will not be any better if it's > restricted to only a subset of the data and if it is only scoped to a > subset of the data then there are extra overheads you'd have beyond what > you normally would have (the same raw value would have the same encoded > value stored per row group). I suppose things like run-length encoding > won't be any less efficient, but it also wouldn't be any more efficient > (with the caveat of a raw value repeating across row groups). > > A different compression for different columns isn't unreasonable, so I > think I could be easily convinced that has benefits (though would require > per-column logic that could slow other things down). > > These are just my thoughts, though. Can you share the design and results > of your benchmark? Have you (or could you) prototyped anything to test it > out? > > Sent from Proton Mail <https://proton.me/mail/home> for iOS > > > On Fri, Mar 22, 2024 at 14:36, Andrei Lazăr <lazarandrei...@gmail.com > <On+Fri,+Mar+22,+2024+at+14:36,+Andrei+Lazăr+%3C%3Ca+href=>> wrote: > > Hi Gang, > > Thanks a lot for getting back to me! > > So the use case I am having is relatively simple: I was playing around with > some data and I wanted to benchmark different compression algorithms in an > effort to speed up data retrieval in a simple Parquet based database that I > am playing around with. Whilst doing so, I've noticed a very large variance > in the performance of the same compression algorithm over different row > groups in my Parquet files. Therefore, I was thinking that the best > compression configuration for my data would be to use a different algorithm > for every column, for every row group in my files. In a real-world > situation, I can see this being used by a database, either when new entries > are inserted into it, or even as a background 'optimizer' job that runs > over existing data. > > How do you feel about this? > > Thank you, > Andrei > > On Thu, 21 Mar 2024 at 02:11, Gang Wu <ust...@gmail.com> wrote: > > > Hi Andrei, > > > > What is your use case? IMHO, exposing this kind of configuration > > will force users to know how will the writer split row groups, which > > does not look simple to me. > > > > Best, > > Gang > > > > On Thu, Mar 21, 2024 at 2:25 AM Andrei Lazăr <lazarandrei...@gmail.com> > > wrote: > > > > > Hi all, > > > > > > I would like proposing adding support for writing a Parquet file with > > > different compression algorithms for every row group. > > > > > > In my understanding, the Parquet format allows this, however it seems > to > > me > > > that there is no way to achieve this from the C++ implementation. > > > > > > Does anyone have any thoughts on this? > > > > > > Thank you, > > > Andrei > > > > > > >
signature.asc
Description: OpenPGP digital signature
