Re: KafkaIO consumer rate

2022-04-11 Thread Alexey Romanenko
Hi Sigalit, Could you try to run your test pipeline with “--experiments=use_deprecated_read” option and see if there is a difference? — Alexey > On 10 Apr 2022, at 21:14, Sigalit Eliazov wrote: > > Hi all > I saw a very low rate when message consuming from kafka in our different > jobs. >

KafkaIO consumer rate

2022-04-10 Thread Sigalit Eliazov
Hi all I saw a very low rate when message consuming from kafka in our different jobs. I order to find the bottleneck i created a very simple pipeline that reads string messages from kafka and just prints the output . The pipeline runs over flink cluster with the following setup: 1 task manager, 3