fx19880617 commented on issue #6804: URL: https://github.com/apache/incubator-pinot/issues/6804#issuecomment-821585761
> > > In columnar database world, encoding is commonly referred to column level compression techniques which play nice with columnar query execution - dictionary encoding, RLE, delta where the true benefit is that query processing can happen faster on compressed columnar data (e.g dictionary encoding) and obviously storage saving is there > > > The purpose of this issue is to not add any new column level encoding. I was thinking of having a separate issue to enhance column level encoding support with RLE, DELTA, PFORDELTA etc. > > > This issue is for supporting additional data compression codecs for raw data which are currently snappy compressed > > > > > > Got it, so we should still make it columnar and do it at block level? Then we still need to add this to schema right? > > Yes this will also be columnar and will be block level although LZ4 supports some form of streaming/frame. I am not sure why do we need to add it to schema? Do you mean configuring it via table config? Cause we want to allow tuning compression per column basis, e.g. column1 in snappy and column2 in lz4 right? This info can be stored: - either inside FieldSpec in schema, - or add a new field in tableConfig, with a map of columns to compression type mapping. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
