Hello William, You have overriden KafkaConsumer's max.poll.records configuration, to return only up to 10 records in single call to poll, so it does that. Fetcher iterates over completed fetch responses and in first one(s) from single topic finds enough of records to satisfy max limit. See https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java#L514
Remove the limit override, and all 20 should be returned in single poll, given all get fetched within poll timeout. Kind regards, Stevo Slavic. On Thu, May 11, 2017 at 8:42 PM, William Yu <w...@unified.com> wrote: > Hi, > > I'm trying to understand the behavior of consumer poll function when > subscribed to multiple topics using a RegEx topic pattern. I was under the > assumption the poll function would pull messages from each of the topics I > was subscribed to, but from a test program I wrote it looks like it will > pull from a given topic until it has been drained. > > Sample program: > https://gist.github.com/wiyu/afb860aacd0382f84601a3512a21e9d1 > > Test : > - 2 Topics: foo and bar > - published 10 msgs to each topic. > > Output: > records to process: 10 > [foo] partition = 2, offset = 83, value = 4 > [foo] partition = 2, offset = 84, value = 8 > [foo] partition = 3, offset = 82, value = 2 > [foo] partition = 3, offset = 83, value = 6 > [foo] partition = 3, offset = 84, value = 10 > [foo] partition = 0, offset = 111, value = 3 > [foo] partition = 0, offset = 112, value = 7 > [foo] partition = 1, offset = 104, value = 1 > [foo] partition = 1, offset = 105, value = 5 > [foo] partition = 1, offset = 106, value = 9 > committed: 10 > records to process: 10 > [bar] partition = 1, offset = 80, value = 2 > [bar] partition = 1, offset = 81, value = 4 > [bar] partition = 1, offset = 82, value = 6 > [bar] partition = 1, offset = 83, value = 8 > [bar] partition = 1, offset = 84, value = 10 > [bar] partition = 0, offset = 80, value = 1 > [bar] partition = 0, offset = 81, value = 3 > [bar] partition = 0, offset = 82, value = 5 > [bar] partition = 0, offset = 83, value = 7 > [bar] partition = 0, offset = 84, value = 9 > > Thanks, > William >
