[
https://issues.apache.org/jira/browse/KAFKA-10133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17182976#comment-17182976
]
Ning Zhang commented on KAFKA-10133:
------------------------------------
[~steveatbat] My experiment seems to show different conclusion.
In order to preserve the compression type from upstream producer, in MM2 config
file, I override the producer config, such as,
{code:java}
<target_cluster_alias>.producer.compression.type = gzip
{code}
Then run this command on the target cluster to verify:
{code:java}
/opt/kafka/bin/kafka-run-class.sh kafka.tools.DumpLogSegments --deep-iteration
--files /kafka/kafka-logs-kafka2-0/primary.test-0/00000000000000000000.log
{code}
Output:
{code:java}
Starting offset: 0
offset: 0 position: 0 CreateTime: 1598249096409 isvalid: true keysize: 4
valuesize: 6 magic: 2 compresscodec: GZIP producerId: -1 producerEpoch: -1
sequence: -1 isTransactional: false headerKeys: []
offset: 1 position: 0 CreateTime: 1598249106432 isvalid: true keysize: 4
valuesize: 6 magic: 2 compresscodec: GZIP producerId: -1 producerEpoch: -1
sequence: -1 isTransactional: false headerKeys: []
offset: 2 position: 0 CreateTime: 1598249158144 isvalid: true keysize: 4
valuesize: 6 magic: 2 compresscodec: GZIP producerId: -1 producerEpoch: -1
sequence: -1 isTransactional: false headerKeys: []
{code}
The output shows that the messages replicated to the target cluster are
compressed with GZIP.
So I am wondering what exact config you use at kafka connect worker level to
make compression work
> Cannot compress messages in destination cluster with MM2
> --------------------------------------------------------
>
> Key: KAFKA-10133
> URL: https://issues.apache.org/jira/browse/KAFKA-10133
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
> Affects Versions: 2.4.0, 2.5.0, 2.4.1
> Environment: kafka 2.5.0 deployed via the strimzi operator 0.18
> Reporter: Steve Jacobs
> Priority: Minor
>
> When configuring mirrormaker2 using kafka connect, it is not possible to
> configure things such that messages are compressed in the destination
> cluster. Dump Log shows that batching is occuring, but no compression. If
> this is possible, then this is a documentation bug, because I can find no
> documentation on how to do this.
> baseOffset: 4208 lastOffset: 4492 count: 285 baseSequence: -1 lastSequence:
> -1 producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 2 isTransactional:
> false isControl: false position: 239371 CreateTime: 1591745894859 size: 16362
> magic: 2 compresscodec: NONE crc: 1811507259 isvalid: true
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
