>From core/src/main/scala/kafka/server/KafkaConfig.scala :
val CompressionTypeDoc = "Specify the final compression type for a given
topic. This configuration accepts the standard compression codecs " +
"('gzip', 'snappy', 'lz4'). It additionally accepts 'uncompressed' which
is equivalent to no compression; and " +
"'producer' which means retain the original compression codec set by the
producer."
You can set the config to 'producer' if you want to retain producer setting.
BTW apostrophe is used in the values.
On Fri, Dec 29, 2017 at 5:59 AM, Sven Ludwig <s_lud...@gmx.de> wrote:
> I checked the latest .log files and the earliest. They are all the same
> with human-readable message payload.
>
>
> I tried setting LZ4 but that leads to a fatal on startup:
>
> [2017-12-29 13:55:15,393] FATAL [KafkaServer id=1] Fatal error during
> KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
> org.apache.kafka.common.config.ConfigException: Invalid value LZ4 for
> configuration compression.type: String must be one of: uncompressed,
> snappy, lz4, gzip, producer
> at org.apache.kafka.common.config.ConfigDef$ValidString.
> ensureValid(ConfigDef.java:897)
> at org.apache.kafka.common.config.ConfigDef.parseValue(
> ConfigDef.java:469)
> at org.apache.kafka.common.config.ConfigDef.parse(
> ConfigDef.java:453)
> at org.apache.kafka.common.config.AbstractConfig.<init>(
> AbstractConfig.java:62)
> at kafka.log.LogConfig.<init>(LogConfig.scala:68)
> at kafka.log.LogManager$.apply(LogManager.scala:783)
> at kafka.server.KafkaServer.startup(KafkaServer.scala:222)
> at io.confluent.support.metrics.SupportedServerStartable.startup(
> SupportedServerStartable.java:112)
> at io.confluent.support.metrics.SupportedKafka.main(
> SupportedKafka.java:58)
>
> So I switched back to lz4 immediately. So no solution so far.
>
>
> If Broker-side compression is applied correctly, does it then recompress
> regarding new incoming messages/batches no matter what compression the
> producer applies? Or are the producer settings somehow relevant even then?
>
>
> Cheers,
> Sven
>
>
> ----------------
> Gesendet: Freitag, 29. Dezember 2017 um 14:45 Uhr
> Von: Manikumar <manikumar.re...@gmail.com>
> An: users@kafka.apache.org
> Betreff: Re: Problem to apply Broker-side lz4 compression even in fresh
> setup
> Is this config added after sending some data? Can you verify the latest
> logs?
> This wont recompress existing messages. Only applicable to new messages.
>
> On Fri, Dec 29, 2017 at 6:59 PM, Ted Yu <yuzhih...@gmail.com> wrote:
>
> > Looking at https://issues.apache.org/jira/browse/KAFKA-5686 , it seems
> you
> > should have specified LZ4.
> >
> > FYI
> >
> > On Fri, Dec 29, 2017 at 5:00 AM, Sven Ludwig <s_lud...@gmx.de> wrote:
> >
> > > Hi,
> > >
> > > we thought we have lz4 applied as broker-side compression on our Kafka
> > > Cluster for storing measurements, but today I looked into the
> individual
> > > .log files and I was able to read all the measurements in plain text by
> > > just using less on the command line. This is for me an indicator that
> > > batches of messages are actually not compressed with lz4. Our setup was
> > > started from scratch i.e. without pre-existing topics, there are no
> > > topic-level overrides and it is based on Confluent Platform 4.0.0
> > >
> > >
> > > 1. Is it perhaps so that we need to care in every Producer that is does
> > > not already compress batches when sending them to the Broker? Up to
> > today I
> > > thought that if the Producer compresses, but the Broker has
> > > compression.type lz4, that the Broker would recompress as lz4?
> > >
> > >
> > > 2. When starting the Broker, in its log-statements it shows the line:
> > > compression.type = lz4
> > > Is this correct, or does the value need to be 'lz4' with apostrophe?
> > >
> > >
> > > 3. Any other hints or possibilities what could be wrong?
> > >
> > >
> > > Generally we would like to enforce lz4 broker-side compression. We do
> not
> > > need to compress data coming from producers, since the network link is
> > not
> > > the problem. We just need to save on disk space.
> > >
> > > Please help us if you can, and have a good new years eve :-)
> > >
> > > Thanks,
> > > Sven
> > >
> >
>
