Re: mirroring Kafka while preserving the order

2017-06-29 Thread James Cheng
MirrorMaker acts as a consumer+producer. So it will consume from the source topic and produce to the destination topic. That means that the destination partition is chosen using the same technique as the normal producer: * if the source record has a key, the key will be hashed and the hash will

Re: mirroring Kafka while preserving the order

2017-06-29 Thread Tom Bentley
I believe so. You need to be careful that the mirror maker producer doesn't reorder messages; in particular if retries > 0 then max.in.flight.requests.per.connection must be 1. If retries=0 then it doesn't matter what max.in.flight.requests.per.connection is. On 29 June 2017 at 05:52, Sunil

mirroring Kafka while preserving the order

2017-06-28 Thread Sunil Parmar
Is it possible to configure mirror maker using message handler to preserve the order of messages in each topic partition. In the particular use case we're looking at both source and destination topics have same number of partitions. Sunil