I just recently noticed KAFKA-658 - Implement "Exact Mirroring" functionality in mirror makerhttps://issues.apache.org/jira/browse/KAFKA-658
If you take that, in combination with the custom message handling in mirrormaker, in combination with KStreams... I think you get an exactly-once stream processor. That could be pretty interesting. I'm sure there are some nuances that make it not quite so simple. In particular, the simple implementation described for KAFKA-658 (ensuring that input offsets and output offsets match) will mean that it would only work for 1-to-1 processing (single input record produces a single output record). And if you have KStreams with a local-state database backed by a kafka changelog, you'd ideally like to synchronize your changelog with your output stream, which may not currently be possible. And of course you probably wouldn't want to use mirrormaker itself, but you'd want to generalize this idea. Anyway, just thought I'd throw that out and see if anyone found the idea interesting. -James ________________________________ This email and any attachments may contain confidential and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments) by others is prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete this email and any attachments. No employee or agent of TiVo Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo Inc. may only be made by a signed written agreement.
