Re: How do I setup Kafka Streams application to have one task per partition?

2017-07-30 Thread Anish Mashankar
Yes, topic names cannot be known before hand and they will keep growing in number. Thanks. The partition grouper appears to be working. On Fri, Jul 28, 2017, 3:28 AM Matthias J. Sax wrote: > Using `PartitionGrouper` is correct. > > As you mentioned correctly, Stream scales via "max number of par

Re: How do I setup Kafka Streams application to have one task per partition?

2017-07-27 Thread Matthias J. Sax
Using `PartitionGrouper` is correct. As you mentioned correctly, Stream scales via "max number of partitions" and thus, be default only create one task for this case. Another way would be, to deploy multiple streams applications each processing a different topic. Of course, for this you will need

How do I setup Kafka Streams application to have one task per partition?

2017-07-25 Thread Anish Mashankar
Hello All, I have more than 100 topics in Kafka with one partition each. These 100 topics are configured through a regex. When running the application, I found that there is only one task that is being spawned as the default partition grouper in Kafka spawns as many tasks as the maximum number of p