Re: kafka producer hangs

have you seen this? https://stackoverflow.com/questions/46649748/kafka-producer-timeoutexception-expiring-1-records jt On Wed, Sep 1, 2021, 10:42 PM Luke Chen wrote: > Hi Владислав, > How do you know "they do not reach the kafka server."? > > We need to know your configs on producer and

Re: [External] Monitoring Consumer

One approach is to have the consumer publish notifications about messages it has processed to a new topic (or other storage mechanism). You may be able to use the admin api, but I don't think it's a standard use case. On Tue, May 25, 2021, 8:21 AM Tauzell, Dave wrote: > I don’t know about

Re: Amazon MSK Feeback

We evaluated confluent cloud and ended up going with https://aiven.io. It's a completely managed solution, running in a private vpc peered with the rest of our aws infrastructure. At the time we evaluated it was substantially less expensive than confluent cloud (~5x). Aiven implemented their own

Re: What happens if commit fails ?

that is possible. review the message delivery semantics section of the documentation for a complete discussion. https://kafka.apache.org/documentation/#semantics On Mon, Jun 8, 2020, 8:27 AM Himanshu Shukla wrote: > Hi All, > Let's say, I pulled 100 messages batch from a consumer, and the

Re: Kafka long running job consumer config best practices and what to do to avoid stuck consumer

the case where we have over-partition a Kafka > topic to reserve some capacity for scaling up in case of burst traffic. > > Regards, > Ali > > On Sat, May 9, 2020 at 11:52 PM Jason Turim wrote: > > > Hi Ali, > > > > You may want to look at using the cons

Re: Kafka long running job consumer config best practices and what to do to avoid stuck consumer

Hi Ali, You may want to look at using the consumer pause / resume api. Its a mechanism that allows you to poll without retrieving new messages. I employed this strategy to effectively handle highly variable workloads by processing them in a background thread. First pause the consumer when

Re: Kafka Streams - Getting exception org.apache.kafka.common.network.InvalidReceiveException exception in cloud

questHeader.parse(RequestHeader.java:126) > at > kafka.network.Processor.$anonfun$processCompletedReceives$1(SocketServer.scala:703) > at > kafka.network.Processor.$anonfun$processCompletedReceives$1$adapted(SocketServer.scala:699) > at scala.collection.Iterator.foreach(Iterator.scala:937) > at scala.co

Am I using the state store appropriately?

​We've started experimenting with Kafka to see if it can be used to aggregate our application data. I think our use case is a match for Kafka streams, but we aren't sure if we are using the tool correctly. The proof of concept we've built seems to be working as designed, I'm not sure that we are

Re: Running kafka in containers

ers > We (heroku) have run databases in containers since 2012, and kafka works > just as well as everything else. So: yes > -- Jason Turim ​Vice President of Software Engineering SignalVine Inc <http://www.signalvine.com> (m) 415-407-6501

Running kafka in containers

Is running a containerized kafka cluster a viable strategy for production?

Question about KIP-107

Hi folks, My team and I are just getting started with Kafka, and I'm looking for information about removing individual items in the log. It led me to this article about implementing GDPR with Kafka . That article links to KIP-107: Add