A couple of remaining questions: - it says in the KIP: "because the metadata topic must be consumed by all instances, we need to assign the topic’s partitions manually and do not commit offsets -- we also need to seekToBeginning() each time we consume the metadata topic)" . How big of a change is allowing a topic to be consumed by all instances?
- for the case when we have a single instance, with intermediate topics etc, could we keep the data that we want to persist in the metadata topic in memory instead? What is the advantage of persisting this data? I understand the need if we have an application with multiple instances on different servers (but I still don't think we need to handle that case). Is there a need to persist data for a single instance? It would help if we enumerate the exact failure scenarios and how persisting the data helps. So I think you convinced me that this metadata is useful in the previous email, now I'm asking if it needs persisting. I'm really trying to avoid having the metadata topic. It's one more topic that needs to be kept around and maintained carefully with all failure cases considered. With EoS around the corner introducing its own internal topic(s), and atomicity when writing to multiple topics, in my mind there is real value if we can have a solution without an extra topic for now. Thanks Eno > On 28 Nov 2016, at 18:47, Matthias J. Sax <matth...@confluent.io> wrote: > > Hi all, > > I want to start a discussion about KIP-95: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-95%3A+Incremental+Batch+Processing+for+Kafka+Streams > > Looking forward to your feedback. > > > -Matthias > >
