Re: Kafka Stream tuning.

2018-02-14 Thread Guozhang Wang
gt; -Original Message- > From: TSANG, Brilly [mailto:brilly.ts...@hk.daiwacm.com] > Sent: Wednesday, February 14, 2018 11:01 AM > To: users@kafka.apache.org > Subject: RE: Kafka Stream tuning. > > Hey Damian and folks, > > I've also tried 1000 and 500 and the perform

RE: Kafka Stream tuning.

2018-02-13 Thread TSANG, Brilly
implementation? Regards, Brilly -Original Message- From: TSANG, Brilly [mailto:brilly.ts...@hk.daiwacm.com] Sent: Wednesday, February 14, 2018 11:01 AM To: users@kafka.apache.org Subject: RE: Kafka Stream tuning. Hey Damian and folks, I've also tried 1000 and 500 and the performance state

RE: Kafka Stream tuning.

2018-02-13 Thread TSANG, Brilly
Kafka Stream tuning. Hi Brilly, My initial guess is that it is the overhead of committing. Commit is synchronous and you have the commit interval set to 50ms. Perhaps try increasing it. Thanks, Damian On Tue, 13 Feb 2018 at 07:49 TSANG, Brilly wrote: > Hi kafka users, > > I creat

Re: Kafka Stream tuning.

2018-02-13 Thread Damian Guy
Hi Brilly, My initial guess is that it is the overhead of committing. Commit is synchronous and you have the commit interval set to 50ms. Perhaps try increasing it. Thanks, Damian On Tue, 13 Feb 2018 at 07:49 TSANG, Brilly wrote: > Hi kafka users, > > I created a filtering stream with the Proc

Kafka Stream tuning.

2018-02-12 Thread TSANG, Brilly
Hi kafka users, I created a filtering stream with the Processor API; input topic that have input rate at ~5 records per millisecond. The filtering function on average takes 0.05milliseconds to complete which in ideal case would translate to (1/0.05) 20 records per millisecond. However, when