GSharayu opened a new issue #6804: URL: https://github.com/apache/incubator-pinot/issues/6804
When the forward index is not dictionary encoded, we have 2 choices: * store the data as is (RAW) store the data snappy compressed - using snappy compression codec library In addition to snappy, we should add support for other compression codecs subject to their availability in Java libraries. Currently by default we use Snappy compression. However, this didn't really give good compression ratio for LTI's free-text data. LZO is known to provide better compression ratio and speed for larger char/varchar data. So, we should explore other options Firstly, we should start with simple test case to compress and uncompress direct byte buffer and do some functional and performance tests. see ZSTD library in Java - https://github.com/luben/zstd-jni Any new ideas/suggestions? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
