Hi James, Yes exactly what we do. Alas as it's bespoke to our internal wrapper ATM so we cannot share this. This is the eco system argument that's with having native headers then we would be able to share these kind of things.
Cheers Mike Sent using OWA for iPhone ________________________________________ From: James Cheng <wushuja...@gmail.com> Sent: Monday, December 5, 2016 8:50:30 AM To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-82 - Add Record Headers > On Dec 2, 2016, at 4:57 PM, Michael Pearce <michael.pea...@ig.com> wrote: > > Hi Jun. > > RE Mirroring, > > [...] > > Lastly around mirroring we have a partionKey field, as the key used for > portioning logic != compaction key all the time but we want to preserve it > for when we mirror so that if source cluster partition count != destination > cluster partition count we can honour the same partitioning logic. > > Michael, Sorry to be off topic about the KIP, but this is genius. I'm totally stealing this idea. We currently require partition counts to be the same on source and destination cluster and do 1-to-1 mapping of partitions (i.e. partition 1 goes to partition 1, partition 2 goes to partition 2, etc) because otherwise mirrormaker would partition based off of the compaction key. Explicitly including the partition key would let us work around that. Does that mean you have a custom partitioner in mirrormaker that uses the partitionKey field? Is it implemented inside a MirrorMakerMessageHandler? Thanks! -James The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
