Hello, I agree with time taken for consumer initialization processes But actually in the test I am taking care of that and I am waiting for the consumer to be initiated and only then starting the producer to discount the initialization delay. So, are there any more processes happening during the poll of consumers for the first few messages?
Thank you On Mon, 13 Dec 2021 at 18:33, Luke Chen <show...@gmail.com> wrote: > Hi Jigar, > > As Liam mentioned, those are necessary consumer initialization processes. > So, I don't think you can speed it up by altering some timeouts/interval > properties. > Is there any reason why you need to care about the initial delay? > If, like you said, the delay won't happen later on, I think the cost will > be amortized. > > > Thank you. > Luke > > > On Mon, Dec 13, 2021 at 4:59 PM Jigar Shah <jigar.shah1...@gmail.com> > wrote: > > > Hello , > > Answering your first mail, indeed I am using consumer groups using > > group.id > > , I must have missed to add it in mentioned properties > > Also, thank you for information regarding the internal processes working > > behind creating a KafkaConsumer. > > I agree that following steps do add latency during initial connection > > creation.But can it be somehow optimised(reduced) ,by altering some > > timeouts/interval properties, could you please suggest those? > > > > Thank you > > > > On Mon, 13 Dec 2021 at 12:05, Liam Clarke-Hutchinson < > lclar...@redhat.com> > > wrote: > > > > > I realise that's a silly question, you must be if you're using auto > > commit. > > > > > > When a consumer starts, it needs to do a few things. > > > > > > 1) Connect to a bootstrap server > > > > > > 2) Join an existing consumer group, or create a new one, if it doesn't > > > exist. This may cause a stop the world rebalance as partitions are > > > reassigned within the group. > > > > > > 3) Acquire metadata - which brokers are the partition leaders for my > > > assigned partitions on? And what offsets am I consuming from? > > > > > > 4) Establish the long lived connections to those brokers. > > > > > > 5) Send fetch requests > > > > > > (I might not have the order correct) > > > > > > So yeah, this is why you're seeing that initial delay before consuming > > > records. > > > > > > Kind regards, > > > > > > Liam Clarke-Hutchinson > > > > > > On Mon, 13 Dec 2021, 7:19 pm Liam Clarke-Hutchinson, < > > lclar...@redhat.com> > > > wrote: > > > > > > > Hi, > > > > > > > > I'm assuming you're using consumer groups? E.g., group.id=X > > > > > > > > Cheers, > > > > > > > > Liam > > > > > > > > On Mon, 13 Dec 2021, 6:30 pm Jigar Shah, <jigar.shah1...@gmail.com> > > > wrote: > > > > > > > >> Hello, > > > >> I am trying to test the latency between message production and > message > > > >> consumption using Java Kafka-Client*(2.7.2)* library. > > > >> The configuration of cluster is 3 KafkaBrokers*(2.7.2, Scala > 2.13)*, 3 > > > >> Zookeeper*(3.5.9)* > > > >> Here is a pattern what I have observed > > > >> Reference: > > > >> ConsumerReadTimeStamp: Timestamp when record received in Kafka > > Consumer > > > >> ProducerTimeStamp: Timestamp added before producer.send record > > > >> RecordTimeStamp: CreateTimeStamp inside the record obtained at > > consumer > > > >> > > > >> [image: kafka1.png] > > > >> > > > >> *For 100 Messages* > > > >> > > > >> *ConsumerReadTimeStamp-ProducerTimeStamp(ms)* > > > >> > > > >> *ConsumerReadTimeStamp-RecordTimeStamp(ms)* > > > >> > > > >> *Average* > > > >> > > > >> *252.56* > > > >> > > > >> *238.85* > > > >> > > > >> *Max* > > > >> > > > >> *2723* > > > >> > > > >> *2016* > > > >> > > > >> *Min* > > > >> > > > >> *125* > > > >> > > > >> *125* > > > >> > > > >> > > > >> On the consumer side it takes too much time for initial few messages > > but > > > >> later on it is quite consistent. > > > >> I have executed the above same test for large number of messages : > > > >> 100,1000,10000,etc. and the pattern seems to be same > > > >> Here are the configurations, mostly using default properties. > > > >> Topic: > > > >> partitions=16 > > > >> min.insync.replica=2 > > > >> replication.factor=3 > > > >> > > > >> > > > >> Consumer: > > > >> security.protocol=PLAINTEXT > > > >> enable.auto.commit=true > > > >> > > > >> > > > >> Producer: > > > >> security.protocol=PLAINTEXT > > > >> compression.type=gzip > > > >> acks=all > > > >> > > > >> > > > >> Is there any reason why there is huge latency at the beginning when > a > > > >> consumer is created please? > > > >> Also please suggest some way to optimise configurations to have some > > > >> better consistent results ? > > > >> > > > >> Thank you in advance for your feedback. > > > >> > > > > > > > > > >
