Dear Kafka, Hi there. I have a question about Kafka Stream parallelism. I know that Kafka Stream parallelism is based on consumer group. Like, if you have 3 partitions source topic you can have maximum 3 consumer instances (or 3 kafka stream instances) at max that will work concurrently. I have 2 scenarios about it and would like to know how it will work?
1. Assume I have a stream pipeline that read from topicA process save to topicB read it back (using through method) process with another logic and save to topicC topicA => transform1 => topicB => transform2 => topicC Supposed that all topics have only 1 partition(for simplicity), Can I have 2 instances of Kafka Stream and expected each one of them to do the transform1 and transform2 balancely ? 2. Is it a good idea to have multiple streaming pipelines in same Kafka Stream application ? Like topicA1 => transformA => topicA2 topicB1 => transformB => topicB2 topicC1 => transformC => topicC2 and like previous one. assume each topic have only 1 partition, can I have 3 Kafka Stream instances and expect each one will take 1 task of each stream pipeline ? Regards, Rapeepat
