Hi Fokko, Thanks for replying, yes sure. The problem we are facing is that with parquet zstd we are not able to control the compression level, we tried setting different compression levels but it doesn't make any difference in the size. We tested/have made sure that we are getting the same compression level in *ZStandardCompressor *as we are setting in the configuration file. Are we missing something? How can we set a different compression level of zstd? Help would be appreciated.
Thanks Falak On Thu, Oct 17, 2019 at 7:47 PM Driesprong, Fokko <fo...@driesprong.frl> wrote: > Hi Manik, > > The supported compression codecs that ship with Parquet are tested and > validated in the CI pipeline. Sometimes there are issues with compressors, > therefore they are not easily pluggable. Feel free to open up a PR to the > project if you believe if there are compressors missing, then we can have a > discussion. > > It is part of the Thrift definition: > https://github.com/apache/parquet-format/blob/37bdba0a18cff18da706a0d353c65e726c8edca6/src/main/thrift/parquet.thrift#L470-L478 > > Hope this clarifies the design decision. > > Cheers, Fokko > > Op di 15 okt. 2019 om 11:52 schreef Manik Singla <smanik...@gmail.com>: > >> Hi >> >> Current java code is not open to use custom compressor. >> I believe mostly read/write is done by same team/company. In that case, >> it >> would be beneficial to add this support that user can plug new compressor >> easily instead of doing local changes which will be prone to uses across >> version upgrades. >> >> Do you guys think it will be worth to add >> >> Regards >> Manik Singla >> +91-9996008893 >> +91-9665639677 >> >> "Life doesn't consist in holding good cards but playing those you hold >> well." >> >
