I've defined several Kafka Connect tasks via the tasks.max property to process a set of topics. Initially I set the partitions on the topics to 1 and partitioned the topics across the tasks programmatically so that each task processed a subset of the topics (or so I thought ...). I then noticed that only 1 of the tasks ever read any Kafka messages and concluded that the topics property defined in connector.properties cannot be split across the tasks in this way.
It then dawned on me that perhaps I ought to be partitioning the topic at creation time so that each task would be assigned a set of partitions across the entire set of topics. However, that seems not to work either - again only 1 task does any work - and this task reads from the same partition for every topic (I have defined 50 partitions and 4 tasks so would expect (naively perhaps) each task to get a dozen or so partitions for each topic). Could some kind soul point out the error of my ways please and tell me how to achieve this properly. Thanks in advance, David
