Looking at https://issues.apache.org/jira/browse/KAFKA-5686 , it seems you should have specified LZ4.
FYI On Fri, Dec 29, 2017 at 5:00 AM, Sven Ludwig <s_lud...@gmx.de> wrote: > Hi, > > we thought we have lz4 applied as broker-side compression on our Kafka > Cluster for storing measurements, but today I looked into the individual > .log files and I was able to read all the measurements in plain text by > just using less on the command line. This is for me an indicator that > batches of messages are actually not compressed with lz4. Our setup was > started from scratch i.e. without pre-existing topics, there are no > topic-level overrides and it is based on Confluent Platform 4.0.0 > > > 1. Is it perhaps so that we need to care in every Producer that is does > not already compress batches when sending them to the Broker? Up to today I > thought that if the Producer compresses, but the Broker has > compression.type lz4, that the Broker would recompress as lz4? > > > 2. When starting the Broker, in its log-statements it shows the line: > compression.type = lz4 > Is this correct, or does the value need to be 'lz4' with apostrophe? > > > 3. Any other hints or possibilities what could be wrong? > > > Generally we would like to enforce lz4 broker-side compression. We do not > need to compress data coming from producers, since the network link is not > the problem. We just need to save on disk space. > > Please help us if you can, and have a good new years eve :-) > > Thanks, > Sven >
