Can someone please clarify my below doubt? The same has been asked on stack overflow also.
https://stackoverflow.com/q/77630586/785523 On Fri, 8 Dec, 2023, 21:33 Debraj Manna, <subharaj.ma...@gmail.com> wrote: > Thanks again. > > Another follow-up question, since max.poll.records has nothing to do with > fetch requests, then is there any gain on number of network calls being > made between consumer & broker if max.poll.records is set to 1 as against > let's say the default 500. > > On Wed, Dec 6, 2023 at 7:21 PM Haruki Okada <ocadar...@gmail.com> wrote: > >> poll-idle-ratio-avg=1.0 doesn't immediately mean fetch throughput problem >> since if processing is very fast, the metric will always be near 1.0. >> >> 2023年12月4日(月) 13:09 Debraj Manna <subharaj.ma...@gmail.com>: >> >> > Thanks for the reply. >> > >> > I read KIP >> > < >> > >> https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=127406453#KIP517:Addconsumermetricstoobserveuserpollbehavior-poll-idle-ratio-avg >> > >. >> > Can you let me know if I observe poll-idle-ration.avg equal to 1.0 then >> > does that mean my fetch.max.bytes or max.partition.fetch.bytes is not >> > enough and I have to increase them? If not what could be the reason that >> > may cause poll-idle-ratio-avg to approach 1.0? >> > >> > >> > Can you let me know what >> > >> > On Sat, 2 Dec, 2023, 07:05 Haruki Okada, <ocadar...@gmail.com> wrote: >> > >> > > Hi. >> > > >> > > `max.poll.records` does nothing with fetch requests (refs: >> > > >> > > >> > >> https://kafka.apache.org/35/documentation.html#consumerconfigs_max.poll.records >> > > ) >> > > >> > > Then, how many records will be returned for single fetch request >> depends >> > on >> > > the partition-leader assignment. (note: we assume follower-fetch is >> not >> > > used here) >> > > If all partition leaders are in the same broker, 40MB (2MB * 20 >> > partition) >> > > will be returned for a single fetch request. >> > > >> > > 2023年11月30日(木) 17:10 Debraj Manna <subharaj.ma...@gmail.com>: >> > > >> > > > The doc states that fetch.max.bytes & max.partition.fetch.bytes >> > > > >> > > > are not absolute maximum. If the first record batch in the first >> > > non-empty >> > > > > partition of the fetch is larger than this limit, the batch will >> > still >> > > be >> > > > > returned to ensure that the consumer can make progress. >> > > > >> > > > >> > > > I am getting a bit confused. >> > > > >> > > > Let's say I have a configuration like below with sufficient >> messages in >> > > > each partition >> > > > >> > > > >> > > > - Partitions in a topic 20 >> > > > - Single message size 2MB >> > > > - Consumers 5 >> > > > - max.poll.records 20 >> > > > - fetch.max.bytes 50 MB >> > > > - max.partition.fetch.bytes 1 MB. >> > > > >> > > > The broker config message.max.bytes and max.message.bytes is set to >> > > default >> > > > 100MB >> > > > >> > > > If the consumer does a poll will it receive 20 records? If yes then >> > there >> > > > is no significance of fetch.max.bytes & max.partition.fetch.bytes >> with >> > > > max.poll.records? >> > > > >> > > > >> > > > - Java Kafka Client - 3.5.1 >> > > > - Kafka Broker - 2.8.1 >> > > > >> > > >> > > >> > > -- >> > > ======================== >> > > Okada Haruki >> > > ocadar...@gmail.com >> > > ======================== >> > > >> > >> >> >> -- >> ======================== >> Okada Haruki >> ocadar...@gmail.com >> ======================== >> >
