Hi,
we thought we have lz4 applied as broker-side compression on our Kafka Cluster
for storing measurements, but today I looked into the individual .log files and
I was able to read all the measurements in plain text by just using less on the
command line. This is for me an indicator that batches of messages are actually
not compressed with lz4. Our setup was started from scratch i.e. without
pre-existing topics, there are no topic-level overrides and it is based on
Confluent Platform 4.0.0
1. Is it perhaps so that we need to care in every Producer that is does not
already compress batches when sending them to the Broker? Up to today I thought
that if the Producer compresses, but the Broker has compression.type lz4, that
the Broker would recompress as lz4?
2. When starting the Broker, in its log-statements it shows the line:
compression.type = lz4
Is this correct, or does the value need to be 'lz4' with apostrophe?
3. Any other hints or possibilities what could be wrong?
Generally we would like to enforce lz4 broker-side compression. We do not need
to compress data coming from producers, since the network link is not the
problem. We just need to save on disk space.
Please help us if you can, and have a good new years eve :-)
Thanks,
Sven
