Is this config added after sending some data? Can you verify the latest logs? This wont recompress existing messages. Only applicable to new messages.
On Fri, Dec 29, 2017 at 6:59 PM, Ted Yu <yuzhih...@gmail.com> wrote: > Looking at https://issues.apache.org/jira/browse/KAFKA-5686 , it seems you > should have specified LZ4. > > FYI > > On Fri, Dec 29, 2017 at 5:00 AM, Sven Ludwig <s_lud...@gmx.de> wrote: > > > Hi, > > > > we thought we have lz4 applied as broker-side compression on our Kafka > > Cluster for storing measurements, but today I looked into the individual > > .log files and I was able to read all the measurements in plain text by > > just using less on the command line. This is for me an indicator that > > batches of messages are actually not compressed with lz4. Our setup was > > started from scratch i.e. without pre-existing topics, there are no > > topic-level overrides and it is based on Confluent Platform 4.0.0 > > > > > > 1. Is it perhaps so that we need to care in every Producer that is does > > not already compress batches when sending them to the Broker? Up to > today I > > thought that if the Producer compresses, but the Broker has > > compression.type lz4, that the Broker would recompress as lz4? > > > > > > 2. When starting the Broker, in its log-statements it shows the line: > > compression.type = lz4 > > Is this correct, or does the value need to be 'lz4' with apostrophe? > > > > > > 3. Any other hints or possibilities what could be wrong? > > > > > > Generally we would like to enforce lz4 broker-side compression. We do not > > need to compress data coming from producers, since the network link is > not > > the problem. We just need to save on disk space. > > > > Please help us if you can, and have a good new years eve :-) > > > > Thanks, > > Sven > > >
