Hi, Can anyone explain me in more detail how Kakfa works with compression ? I've read the doc but it's not all clear to me.
- There are compression settings on the broker, the topic of a broker, a producer. Are they all the same setting and one takes precedence on another ? - Is there a difference between compression for transfer and compression for storage ? - Is the compression done per message or per topic ? For example, my message which will transit from / to kafka are in a JSON format (so with similar structure). will compression compress a chunk of several message (which I guess would be more efficient in my case considering my bottle neck is the transport link) or will it compress every message separately ? The documentation would be great if illustrated with few examples like : Broker has compression type GZIP Topic-A has compression Snappy Topic-B has no compression setting Producer-X has compression GZIP Producer-Y has no compression settings. explain what happens (on the wire, and on disk) when X Produces to A X Produces to B Y Produces to A Y Produces to B + explain if some settings n-uple are nonsense / imcompatible / could result in errors. Thank you. -- The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer.
