Re: max.task.idle.ms behavior

[Matthias J. Sax]

The parameter applies too all three topics (input, intermediate,
repartitions topics) and it's a global config.

About the blocking behavior:

If one partitions becomes empty, all other partitions are paused() and
Streams only poll() for the empty partition. If no data is returned
within the timeout, processing of enforced for this task for all partitions.

The task will stay in the "enforced processing state" until all
partitions deliver data again (at the same time).

If a second partition becomes empty, no additional delay is applied.

Hope this answers your question.


-Matthias


On 1/15/19 2:07 AM, Peter Levart wrote:
> Another question about this parameter.
> 
> Does that parameter apply just to input topics to the KafkaStreams
> topology or also to intermediate (repartitioning) topics or to
> intermediate topics configured with KStream.through() directive?
> 
> Is it possible to control the behavior on a per-topic basis?
> 
> Thanks,
> 
> Peter
> 
> On 1/15/19 11:00 AM, Peter Levart wrote:
>> Hello!
>>
>> I'm trying to understand the behavior of Kafka Streams consumers with
>> regards to max.task.idle.ms configuration parameter (default 0). The
>> documentation says:
>>
>> max.task.idle.ms     Medium     Maximum amount of time a stream task
>> will stay idle when not all of its partition buffers contain records.
>>
>> Suppose an input topic to a streams application has multiple
>> partitions and that traffic arrives to this input topic in bursts. At
>> first all partitions will be filled with records and all tasks will be
>> busy. Then some partition will get drained first. The task with
>> drained partition will bet paused for max.task.idle.ms time.
>>
>> Question: How often will the task with drained partition pause for
>> max.task.idle.ms time due to missing records in a partition?
>>
>> 1. Before every record?
>> 2. At each KafkaConsumer.poll() that returns any records?
>> 3. Just the 1st time some partition is detected to have no polled
>> records, but then this partition will get "blacklisted" from further
>> checks for pausing until it again receives a record.
>>
>> For example, if the task is assigned N partitions, what is the max. #
>> of times the task will pause for max.task.idle.ms time when the input
>> burst is over:
>>
>> The wishful answer is N-1 times. Is this so?
>>
>> Regards, Peter
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: [ANNOUNCE] New Committer: Vahid Hashemian

[Dongjin Lee]

Congratulations, Vahid!!

On Wed, Jan 16, 2019 at 3:32 PM Dong Lin  wrote:

> Congratulations Vahid!
>
> On Tue, Jan 15, 2019 at 2:45 PM Jason Gustafson 
> wrote:
>
> > Hi All,
> >
> > The PMC for Apache Kafka has invited Vahid Hashemian as a project
> > committer and
> > we are
> > pleased to announce that he has accepted!
> >
> > Vahid has made numerous contributions to the Kafka community over the
> past
> > few years. He has authored 13 KIPs with core improvements to the consumer
> > and the tooling around it. He has also contributed nearly 100 patches
> > affecting all parts of the codebase. Additionally, Vahid puts a lot of
> > effort into community engagement, helping others on the mail lists and
> > sharing his experience at conferences and meetups.
> >
> > We appreciate the contributions and we are looking forward to more.
> > Congrats Vahid!
> >
> > Jason, on behalf of the Apache Kafka PMC
> >
>


-- 
*Dongjin Lee*

*A hitchhiker in the mathematical world.*
*github:  github.com/dongjinleekr
linkedin: kr.linkedin.com/in/dongjinleekr
speakerdeck: speakerdeck.com/dongjin
*

Re: [ANNOUNCE] New Committer: Vahid Hashemian

[Jan Filipiak]

Congratz!

On 15.01.2019 23:44, Jason Gustafson wrote:
> Hi All,
>
> The PMC for Apache Kafka has invited Vahid Hashemian as a project committer 
> and
> we are
> pleased to announce that he has accepted!
>
> Vahid has made numerous contributions to the Kafka community over the past
> few years. He has authored 13 KIPs with core improvements to the consumer
> and the tooling around it. He has also contributed nearly 100 patches
> affecting all parts of the codebase. Additionally, Vahid puts a lot of
> effort into community engagement, helping others on the mail lists and
> sharing his experience at conferences and meetups.
>
> We appreciate the contributions and we are looking forward to more.
> Congrats Vahid!
>
> Jason, on behalf of the Apache Kafka PMC
>

Re: [ANNOUNCE] New Committer: Vahid Hashemian

[Suman B N]

Congrats Vahid!

On Wed, Jan 16, 2019 at 12:02 PM Dong Lin  wrote:

> Congratulations Vahid!
>
> On Tue, Jan 15, 2019 at 2:45 PM Jason Gustafson 
> wrote:
>
> > Hi All,
> >
> > The PMC for Apache Kafka has invited Vahid Hashemian as a project
> > committer and
> > we are
> > pleased to announce that he has accepted!
> >
> > Vahid has made numerous contributions to the Kafka community over the
> past
> > few years. He has authored 13 KIPs with core improvements to the consumer
> > and the tooling around it. He has also contributed nearly 100 patches
> > affecting all parts of the codebase. Additionally, Vahid puts a lot of
> > effort into community engagement, helping others on the mail lists and
> > sharing his experience at conferences and meetups.
> >
> > We appreciate the contributions and we are looking forward to more.
> > Congrats Vahid!
> >
> > Jason, on behalf of the Apache Kafka PMC
> >
>


-- 
*Suman*
*OlaCabs*

Re: [ANNOUNCE] New Committer: Vahid Hashemian

[Dong Lin]

Congratulations Vahid!

On Tue, Jan 15, 2019 at 2:45 PM Jason Gustafson  wrote:

> Hi All,
>
> The PMC for Apache Kafka has invited Vahid Hashemian as a project
> committer and
> we are
> pleased to announce that he has accepted!
>
> Vahid has made numerous contributions to the Kafka community over the past
> few years. He has authored 13 KIPs with core improvements to the consumer
> and the tooling around it. He has also contributed nearly 100 patches
> affecting all parts of the codebase. Additionally, Vahid puts a lot of
> effort into community engagement, helping others on the mail lists and
> sharing his experience at conferences and meetups.
>
> We appreciate the contributions and we are looking forward to more.
> Congrats Vahid!
>
> Jason, on behalf of the Apache Kafka PMC
>

Re: [ANNOUNCE] New Committer: Vahid Hashemian

[Alex D]

Congrats, Vahid!

On Wed, 16 Jan 2019, 05:54 Gwen Shapira  Congrats, Vahid. Thank you for all your contribution!
>
> On Tue, Jan 15, 2019, 2:45 PM Jason Gustafson 
> > Hi All,
> >
> > The PMC for Apache Kafka has invited Vahid Hashemian as a project
> > committer and
> > we are
> > pleased to announce that he has accepted!
> >
> > Vahid has made numerous contributions to the Kafka community over the
> past
> > few years. He has authored 13 KIPs with core improvements to the consumer
> > and the tooling around it. He has also contributed nearly 100 patches
> > affecting all parts of the codebase. Additionally, Vahid puts a lot of
> > effort into community engagement, helping others on the mail lists and
> > sharing his experience at conferences and meetups.
> >
> > We appreciate the contributions and we are looking forward to more.
> > Congrats Vahid!
> >
> > Jason, on behalf of the Apache Kafka PMC
> >
>

Re: [ANNOUNCE] New Committer: Vahid Hashemian

[Gwen Shapira]

Congrats, Vahid. Thank you for all your contribution!

On Tue, Jan 15, 2019, 2:45 PM Jason Gustafson  Hi All,
>
> The PMC for Apache Kafka has invited Vahid Hashemian as a project
> committer and
> we are
> pleased to announce that he has accepted!
>
> Vahid has made numerous contributions to the Kafka community over the past
> few years. He has authored 13 KIPs with core improvements to the consumer
> and the tooling around it. He has also contributed nearly 100 patches
> affecting all parts of the codebase. Additionally, Vahid puts a lot of
> effort into community engagement, helping others on the mail lists and
> sharing his experience at conferences and meetups.
>
> We appreciate the contributions and we are looking forward to more.
> Congrats Vahid!
>
> Jason, on behalf of the Apache Kafka PMC
>

Re: [ANNOUNCE] New Committer: Vahid Hashemian

[hacker win7]

Congrats!

> On Jan 16, 2019, at 10:08, Srinivas Reddy  wrote:
> 
> Congratulation Vahid!!!
> 
> 
>> On 16 Jan 2019, at 9:58 AM, Manikumar  wrote:
>> 
>> Congrats, Vahid!
>> 
>> On Wed 16 Jan, 2019, 6:53 AM Ismael Juma > 
>>> Congratulations Vahid!
>>> 
>>> On Tue, Jan 15, 2019, 2:45 PM Jason Gustafson >> 
 Hi All,
 
 The PMC for Apache Kafka has invited Vahid Hashemian as a project
 committer and
 we are
 pleased to announce that he has accepted!
 
 Vahid has made numerous contributions to the Kafka community over the
>>> past
 few years. He has authored 13 KIPs with core improvements to the consumer
 and the tooling around it. He has also contributed nearly 100 patches
 affecting all parts of the codebase. Additionally, Vahid puts a lot of
 effort into community engagement, helping others on the mail lists and
 sharing his experience at conferences and meetups.
 
 We appreciate the contributions and we are looking forward to more.
 Congrats Vahid!
 
 Jason, on behalf of the Apache Kafka PMC
 
>>> 
> 

Re: [ANNOUNCE] New Committer: Vahid Hashemian

[Srinivas Reddy]

Congratulation Vahid!!!


> On 16 Jan 2019, at 9:58 AM, Manikumar  wrote:
> 
> Congrats, Vahid!
> 
> On Wed 16 Jan, 2019, 6:53 AM Ismael Juma  
>> Congratulations Vahid!
>> 
>> On Tue, Jan 15, 2019, 2:45 PM Jason Gustafson > 
>>> Hi All,
>>> 
>>> The PMC for Apache Kafka has invited Vahid Hashemian as a project
>>> committer and
>>> we are
>>> pleased to announce that he has accepted!
>>> 
>>> Vahid has made numerous contributions to the Kafka community over the
>> past
>>> few years. He has authored 13 KIPs with core improvements to the consumer
>>> and the tooling around it. He has also contributed nearly 100 patches
>>> affecting all parts of the codebase. Additionally, Vahid puts a lot of
>>> effort into community engagement, helping others on the mail lists and
>>> sharing his experience at conferences and meetups.
>>> 
>>> We appreciate the contributions and we are looking forward to more.
>>> Congrats Vahid!
>>> 
>>> Jason, on behalf of the Apache Kafka PMC
>>> 
>> 

Re: [ANNOUNCE] New Committer: Vahid Hashemian

[Manikumar]

Congrats, Vahid!

On Wed 16 Jan, 2019, 6:53 AM Ismael Juma  Congratulations Vahid!
>
> On Tue, Jan 15, 2019, 2:45 PM Jason Gustafson 
> > Hi All,
> >
> > The PMC for Apache Kafka has invited Vahid Hashemian as a project
> > committer and
> > we are
> > pleased to announce that he has accepted!
> >
> > Vahid has made numerous contributions to the Kafka community over the
> past
> > few years. He has authored 13 KIPs with core improvements to the consumer
> > and the tooling around it. He has also contributed nearly 100 patches
> > affecting all parts of the codebase. Additionally, Vahid puts a lot of
> > effort into community engagement, helping others on the mail lists and
> > sharing his experience at conferences and meetups.
> >
> > We appreciate the contributions and we are looking forward to more.
> > Congrats Vahid!
> >
> > Jason, on behalf of the Apache Kafka PMC
> >
>

Re: [ANNOUNCE] New Committer: Vahid Hashemian

[Ismael Juma]

Congratulations Vahid!

On Tue, Jan 15, 2019, 2:45 PM Jason Gustafson  Hi All,
>
> The PMC for Apache Kafka has invited Vahid Hashemian as a project
> committer and
> we are
> pleased to announce that he has accepted!
>
> Vahid has made numerous contributions to the Kafka community over the past
> few years. He has authored 13 KIPs with core improvements to the consumer
> and the tooling around it. He has also contributed nearly 100 patches
> affecting all parts of the codebase. Additionally, Vahid puts a lot of
> effort into community engagement, helping others on the mail lists and
> sharing his experience at conferences and meetups.
>
> We appreciate the contributions and we are looking forward to more.
> Congrats Vahid!
>
> Jason, on behalf of the Apache Kafka PMC
>

Re: [ANNOUNCE] New Committer: Vahid Hashemian

[Martin Gainty]

Բարի գալուստ

(Bari Galust)


From: Satish Duggana 
Sent: Tuesday, January 15, 2019 7:54 PM
To: dev
Cc: Kafka Users
Subject: Re: [ANNOUNCE] New Committer: Vahid Hashemian

Congratulations Vahid!

On Tue, Jan 15, 2019 at 5:47 PM Yishun Guan  wrote:
>
> Congratulations!
>
> On Tue, Jan 15, 2019, 16:46 James Cheng  wrote:
>
> > Congrats, Vahid!!
> >
> > -James
> >
> > > On Jan 15, 2019, at 2:44 PM, Jason Gustafson  wrote:
> > >
> > > Hi All,
> > >
> > > The PMC for Apache Kafka has invited Vahid Hashemian as a project
> > committer and
> > > we are
> > > pleased to announce that he has accepted!
> > >
> > > Vahid has made numerous contributions to the Kafka community over the
> > past
> > > few years. He has authored 13 KIPs with core improvements to the consumer
> > > and the tooling around it. He has also contributed nearly 100 patches
> > > affecting all parts of the codebase. Additionally, Vahid puts a lot of
> > > effort into community engagement, helping others on the mail lists and
> > > sharing his experience at conferences and meetups.
> > >
> > > We appreciate the contributions and we are looking forward to more.
> > > Congrats Vahid!
> > >
> > > Jason, on behalf of the Apache Kafka PMC
> >
> >

Re: [ANNOUNCE] New Committer: Vahid Hashemian

[Satish Duggana]

Congratulations Vahid!

On Tue, Jan 15, 2019 at 5:47 PM Yishun Guan  wrote:
>
> Congratulations!
>
> On Tue, Jan 15, 2019, 16:46 James Cheng  wrote:
>
> > Congrats, Vahid!!
> >
> > -James
> >
> > > On Jan 15, 2019, at 2:44 PM, Jason Gustafson  wrote:
> > >
> > > Hi All,
> > >
> > > The PMC for Apache Kafka has invited Vahid Hashemian as a project
> > committer and
> > > we are
> > > pleased to announce that he has accepted!
> > >
> > > Vahid has made numerous contributions to the Kafka community over the
> > past
> > > few years. He has authored 13 KIPs with core improvements to the consumer
> > > and the tooling around it. He has also contributed nearly 100 patches
> > > affecting all parts of the codebase. Additionally, Vahid puts a lot of
> > > effort into community engagement, helping others on the mail lists and
> > > sharing his experience at conferences and meetups.
> > >
> > > We appreciate the contributions and we are looking forward to more.
> > > Congrats Vahid!
> > >
> > > Jason, on behalf of the Apache Kafka PMC
> >
> >

Re: [ANNOUNCE] New Committer: Vahid Hashemian

[Yishun Guan]

Congratulations!

On Tue, Jan 15, 2019, 16:46 James Cheng  wrote:

> Congrats, Vahid!!
>
> -James
>
> > On Jan 15, 2019, at 2:44 PM, Jason Gustafson  wrote:
> >
> > Hi All,
> >
> > The PMC for Apache Kafka has invited Vahid Hashemian as a project
> committer and
> > we are
> > pleased to announce that he has accepted!
> >
> > Vahid has made numerous contributions to the Kafka community over the
> past
> > few years. He has authored 13 KIPs with core improvements to the consumer
> > and the tooling around it. He has also contributed nearly 100 patches
> > affecting all parts of the codebase. Additionally, Vahid puts a lot of
> > effort into community engagement, helping others on the mail lists and
> > sharing his experience at conferences and meetups.
> >
> > We appreciate the contributions and we are looking forward to more.
> > Congrats Vahid!
> >
> > Jason, on behalf of the Apache Kafka PMC
>
>

Re: [ANNOUNCE] New Committer: Vahid Hashemian

[James Cheng]

Congrats, Vahid!!

-James

> On Jan 15, 2019, at 2:44 PM, Jason Gustafson  wrote:
> 
> Hi All,
> 
> The PMC for Apache Kafka has invited Vahid Hashemian as a project committer 
> and
> we are
> pleased to announce that he has accepted!
> 
> Vahid has made numerous contributions to the Kafka community over the past
> few years. He has authored 13 KIPs with core improvements to the consumer
> and the tooling around it. He has also contributed nearly 100 patches
> affecting all parts of the codebase. Additionally, Vahid puts a lot of
> effort into community engagement, helping others on the mail lists and
> sharing his experience at conferences and meetups.
> 
> We appreciate the contributions and we are looking forward to more.
> Congrats Vahid!
> 
> Jason, on behalf of the Apache Kafka PMC

Re: [ANNOUNCE] New Committer: Vahid Hashemian

[Jun Rao]

Congratulations, Vahid.

Thanks,

Jun

On Tue, Jan 15, 2019 at 2:45 PM Jason Gustafson  wrote:

> Hi All,
>
> The PMC for Apache Kafka has invited Vahid Hashemian as a project
> committer and
> we are
> pleased to announce that he has accepted!
>
> Vahid has made numerous contributions to the Kafka community over the past
> few years. He has authored 13 KIPs with core improvements to the consumer
> and the tooling around it. He has also contributed nearly 100 patches
> affecting all parts of the codebase. Additionally, Vahid puts a lot of
> effort into community engagement, helping others on the mail lists and
> sharing his experience at conferences and meetups.
>
> We appreciate the contributions and we are looking forward to more.
> Congrats Vahid!
>
> Jason, on behalf of the Apache Kafka PMC
>

Re: [ANNOUNCE] New Committer: Vahid Hashemian

[Guozhang Wang]

Congratulations Vahid!

Guozhang

On Tue, Jan 15, 2019 at 2:58 PM Bill Bejeck  wrote:

> Congrats!
>
> On Tue, Jan 15, 2019 at 5:50 PM Rajini Sivaram 
> wrote:
>
> > Congratulations, Vahid! Well deserved!!
> >
> > Regards,
> >
> > Rajini
> >
> > On Tue, Jan 15, 2019 at 10:45 PM Jason Gustafson 
> > wrote:
> >
> > > Hi All,
> > >
> > > The PMC for Apache Kafka has invited Vahid Hashemian as a project
> > > committer and
> > > we are
> > > pleased to announce that he has accepted!
> > >
> > > Vahid has made numerous contributions to the Kafka community over the
> > past
> > > few years. He has authored 13 KIPs with core improvements to the
> consumer
> > > and the tooling around it. He has also contributed nearly 100 patches
> > > affecting all parts of the codebase. Additionally, Vahid puts a lot of
> > > effort into community engagement, helping others on the mail lists and
> > > sharing his experience at conferences and meetups.
> > >
> > > We appreciate the contributions and we are looking forward to more.
> > > Congrats Vahid!
> > >
> > > Jason, on behalf of the Apache Kafka PMC
> > >
> >
>


-- 
-- Guozhang

Re: [ANNOUNCE] New Committer: Vahid Hashemian

[Bill Bejeck]

Congrats!

On Tue, Jan 15, 2019 at 5:50 PM Rajini Sivaram 
wrote:

> Congratulations, Vahid! Well deserved!!
>
> Regards,
>
> Rajini
>
> On Tue, Jan 15, 2019 at 10:45 PM Jason Gustafson 
> wrote:
>
> > Hi All,
> >
> > The PMC for Apache Kafka has invited Vahid Hashemian as a project
> > committer and
> > we are
> > pleased to announce that he has accepted!
> >
> > Vahid has made numerous contributions to the Kafka community over the
> past
> > few years. He has authored 13 KIPs with core improvements to the consumer
> > and the tooling around it. He has also contributed nearly 100 patches
> > affecting all parts of the codebase. Additionally, Vahid puts a lot of
> > effort into community engagement, helping others on the mail lists and
> > sharing his experience at conferences and meetups.
> >
> > We appreciate the contributions and we are looking forward to more.
> > Congrats Vahid!
> >
> > Jason, on behalf of the Apache Kafka PMC
> >
>

Re: [ANNOUNCE] New Committer: Vahid Hashemian

[Rajini Sivaram]

Congratulations, Vahid! Well deserved!!

Regards,

Rajini

On Tue, Jan 15, 2019 at 10:45 PM Jason Gustafson  wrote:

> Hi All,
>
> The PMC for Apache Kafka has invited Vahid Hashemian as a project
> committer and
> we are
> pleased to announce that he has accepted!
>
> Vahid has made numerous contributions to the Kafka community over the past
> few years. He has authored 13 KIPs with core improvements to the consumer
> and the tooling around it. He has also contributed nearly 100 patches
> affecting all parts of the codebase. Additionally, Vahid puts a lot of
> effort into community engagement, helping others on the mail lists and
> sharing his experience at conferences and meetups.
>
> We appreciate the contributions and we are looking forward to more.
> Congrats Vahid!
>
> Jason, on behalf of the Apache Kafka PMC
>

[ANNOUNCE] New Committer: Vahid Hashemian

[Jason Gustafson]

Hi All,

The PMC for Apache Kafka has invited Vahid Hashemian as a project committer and
we are
pleased to announce that he has accepted!

Vahid has made numerous contributions to the Kafka community over the past
few years. He has authored 13 KIPs with core improvements to the consumer
and the tooling around it. He has also contributed nearly 100 patches
affecting all parts of the codebase. Additionally, Vahid puts a lot of
effort into community engagement, helping others on the mail lists and
sharing his experience at conferences and meetups.

We appreciate the contributions and we are looking forward to more.
Congrats Vahid!

Jason, on behalf of the Apache Kafka PMC

Subscribe me, please

[Javier Arias Losada]

Question on librdkafka.Dll rd_kafka_topic_result_error

[Super User]

The number on threads grows as application continue on pooling message.
I got almost 1000 threads hanging and a thousands of connection established.
Here is the stack for a thread:
0x
ntdll.dll!NtWaitForSingleObject+0x14
mswsock.dll!Tcpip4_WSHOpenSocket2+0x332
mswsock.dll!sethostname+0x3081
mswsock.dll!Tcpip4_WSHStringToAddress+0x2424
ws2_32.dll!WSAIoctl+0x18f
ws2_32.dll!WSAPoll+0x1ea
librdkafka.DLL!rd_kafka_topic_partition_available+0x2701
librdkafka.DLL!rd_kafka_topic_partition_available+0x26a5
librdkafka.DLL!rd_kafka_wait_destroyed+0x86ea
librdkafka.DLL!rd_kafka_wait_destroyed+0x35df
librdkafka.DLL!rd_kafka_wait_destroyed+0x8dec
librdkafka.DLL!rd_kafka_topic_result_error+0x452b
KERNEL32.DLL!BaseThreadInitThunk+0x14
ntdll.dll!RtlUserThreadStart+0x21

Any idea of why that is happening?
Best regards,
Rus

Does Kafka batch when using broker-side compression?

[Sven Ludwig]

Hi,

does a Kafka broker batch messages together in order to improve compression 
when using broker-side compression only?

If so, what are the prerequisites to get that behavior? Perhaps a batching of 
messages at a producer might be beneficial or even mandatory in order to get 
batches in the log?

In my scenario, there is no compression on the producer, and it is also not 
planned to introduce it.

Cheers
Sven

Re: max.task.idle.ms behavior

[Peter Levart]

Another question about this parameter.

Does that parameter apply just to input topics to the KafkaStreams 
topology or also to intermediate (repartitioning) topics or to 
intermediate topics configured with KStream.through() directive?


Is it possible to control the behavior on a per-topic basis?

Thanks,

Peter

On 1/15/19 11:00 AM, Peter Levart wrote:

Hello!

I'm trying to understand the behavior of Kafka Streams consumers with 
regards to max.task.idle.ms configuration parameter (default 0). The 
documentation says:


max.task.idle.ms     Medium     Maximum amount of time a stream task 
will stay idle when not all of its partition buffers contain records.


Suppose an input topic to a streams application has multiple 
partitions and that traffic arrives to this input topic in bursts. At 
first all partitions will be filled with records and all tasks will be 
busy. Then some partition will get drained first. The task with 
drained partition will bet paused for max.task.idle.ms time.


Question: How often will the task with drained partition pause for 
max.task.idle.ms time due to missing records in a partition?


1. Before every record?
2. At each KafkaConsumer.poll() that returns any records?
3. Just the 1st time some partition is detected to have no polled 
records, but then this partition will get "blacklisted" from further 
checks for pausing until it again receives a record.


For example, if the task is assigned N partitions, what is the max. # 
of times the task will pause for max.task.idle.ms time when the input 
burst is over:


The wishful answer is N-1 times. Is this so?

Regards, Peter

max.task.idle.ms behavior

[Peter Levart]

Hello!

I'm trying to understand the behavior of Kafka Streams consumers with 
regards to max.task.idle.ms configuration parameter (default 0). The 
documentation says:


max.task.idle.ms     Medium     Maximum amount of time a stream task 
will stay idle when not all of its partition buffers contain records.


Suppose an input topic to a streams application has multiple partitions 
and that traffic arrives to this input topic in bursts. At first all 
partitions will be filled with records and all tasks will be busy. Then 
some partition will get drained first. The task with drained partition 
will bet paused for max.task.idle.ms time.


Question: How often will the task with drained partition pause for 
max.task.idle.ms time due to missing records in a partition?


1. Before every record?
2. At each KafkaConsumer.poll() that returns any records?
3. Just the 1st time some partition is detected to have no polled 
records, but then this partition will get "blacklisted" from further 
checks for pausing until it again receives a record.


For example, if the task is assigned N partitions, what is the max. # of 
times the task will pause for max.task.idle.ms time when the input burst 
is over:


The wishful answer is N-1 times. Is this so?

Regards, Peter