Re: monitoring length of queues

2022-01-11 Thread Nhat Bui
Florin, What may be of use is seeing the "consumer lag”. This is the offset position of the latest message consumed by the consumer group, per partition, vs. the last offset of the topic. So if a lag is 3, that means there are 3 unconsumed messages by the consumer group. I’m unsure where it

Re: Questions around static membership partition assignment

2022-01-11 Thread Luke Chen
Hi Jun The goal of static membership, is to hold the rebalance when there's consumer dropped (before session timeout). For K8s, it's good because when the pods are broken (or during upgrade), it'll kill the pod and bring a new one up to replace the old one. In this case, we don't want the

Questions around static membership partition assignment

2022-01-11 Thread jun aoki
Hi kafka experts, My understanding of static membership is that assuming kubernetes, for example, can provide a fixed number of healthy pods almost always, so that kafka doesn't have to do any rebalancing. It leads me to think, if the starting point is partition assignment being unbalanced (say

Kafka streams usecase

2022-01-11 Thread pradeep s
Hi , I have a requirement to stream item details to specific destinations . There are three different kafka streams , one for item info, second for item price and promotions and third for item availability . I want to join all these info and produce a single message containing item,price and

monitoring length of queues

2022-01-11 Thread Florin Andrei
We have several producers and several consumers connected via Kafka - basically, it's batch processing jobs that are created on demand, placed on Kafka in several queues, and the batch processors pick them up via Kafka and process them one by one. I want to visualize and monitor the lengths