Hello!
I was looking into compression for some WAN mirroring and did some
tcpdumping.
Our producer is using snappy and I think I can see that in the network
traffic. While the outer part of each message is readable, the content
looks compressed to me with mostly nonprintable characters. However upon
consuming by the MirrorMaker complete message contents are readable in
the dump. To clarify:
producer --(snappy)--> broker --(uncompressed, WAN)--> MirrorMaker(consumer)
From what I read about kafka (end-to-end) compression that shouldn't
even be possible? As far as I can see compression can only be configured
at the producer.
How can I be sure especially the communication via WAN is compressed?
Does anybody have experience mirroring via WAN and can give some hints
on the MM configuration as well as OS Network tuning regarding kafka?
Also the option shallow.iterator.enable does not seem to be documented.
Regarding the above, could that help (although I don't suppose so,
because the messages already have to leave the broker compressed)?
Cheers
Jörg
