OffsetsForTimes returns null when it shouldn't

2020-12-14 Thread Gong, Yifei
Hi team, I found that when I seek to a recent timestamp using OffsetsForTimes method: https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L2078 It returns me offsets for all partitions. However, if I seek an earlier timestamp, it

ProducerFencedException

2020-12-14 Thread Indu V
Hi Team, I am facing an issue in a single Kafka environment trying to connect from multiple application instances. org.apache.kafka.common.KafkaException: Cannot perform send because at least one previous transactional or idempotent request has failed with errors. Caused by:

Kafka support Arm CI

2020-12-14 Thread Peng Lei
Hi Kafka: Recently we met a problem that we have to test and run Kafka on ARM arch. While searching the Kafka community, I didn’t find an official ARM release version. Since Kafka is made by Java and Scala language which can be run cross-platform usually, I think Kafka can be built and run

Re: [VOTE] 2.7.0 RC5

2020-12-14 Thread John Roesler
Thanks for this release, Bill, I ran though the quickstart (just the zk, broker, and console clients part), verified the signatures, and also built and ran the tests. I'm +1 (binding). Thanks, -John On Mon, 2020-12-14 at 14:58 -0800, Guozhang Wang wrote: > I checked the docs and ran unit

Re: Kafka Streams Optimizations

2020-12-14 Thread Navneeth Krishnan
Thanks Guozhang for the response. Appreciate the help. Regards, Navneeth On Mon, Dec 14, 2020 at 2:44 PM Guozhang Wang wrote: > Hello Navneeth, > > Please find answers to your questions below. > > On Sun, Dec 6, 2020 at 10:33 PM Navneeth Krishnan < > reachnavnee...@gmail.com> > wrote: > > > Hi

Re: Kafka Streams - Source Partition Assignment Issue

2020-12-14 Thread Navneeth Krishnan
Thanks Guozhang for the suggestion. We are using kafka 2.3.0 and the app.id is set to the same value. Bouncing off instances work for a small period of time but when auto scaling happens it skews it up again. This is becoming a big bottleneck for us since only one node is now trying to handle

Re: Handling "uneven" network partitions

2020-12-14 Thread Guozhang Wang
Hello Stig, I think there's an ordering of the events here, e.g.: T0: network partition happens. T1: record-1 received at the leader, at this time the ISR is still 3. Leader will accept this record and wait for it to be replicated. T100: lafter some elapsed time the leader decides to kick other

Re: [VOTE] 2.7.0 RC5

2020-12-14 Thread Guozhang Wang
I checked the docs and ran unit tests, no red flags found. +1. On Fri, Dec 11, 2020 at 5:45 AM Bill Bejeck wrote: > Updated with link to successful Jenkins build. > > * Successful Jenkins builds for the 2.7 branch: > Unit/integration tests: > >

Re: Kafka Streams - Source Partition Assignment Issue

2020-12-14 Thread Guozhang Wang
Kafka Streams should evenly distribute the partitions, but there are some issues in old versions of Kafka that you may be observing. To verify if it is a transient issue or it is permanently, I'd suggest you try: 1) bounce the instances that have no partitions assigned (not bounce them all

Re: Kafka Streams Optimizations

2020-12-14 Thread Guozhang Wang
Hello Navneeth, Please find answers to your questions below. On Sun, Dec 6, 2020 at 10:33 PM Navneeth Krishnan wrote: > Hi All, > > I have been working on moving an application to kafka streams and I have > the following questions. > > 1. We were planning to use an EFS mount to share rocksdb

Kafka Streams - Source Partition Assignment Issue

2020-12-14 Thread Navneeth Krishnan
Hi All, How does kafka streams partition assignment work for sources? I have a stream application reading from a topic which has 24 partitions. There are 6 application containers with 4 stream tasks in each container running but only 2 instances are assigned with partitions and even within the

Re: Kafka Streams Optimizations

2020-12-14 Thread Navneeth Krishnan
Hi All, Any advice would really help. Thanks, Navneeth On Sun, Dec 6, 2020 at 10:32 PM Navneeth Krishnan wrote: > Hi All, > > I have been working on moving an application to kafka streams and I have > the following questions. > > 1. We were planning to use an EFS mount to share rocksdb data

Re: Announcing #TwelveDaysOfSMT

2020-12-14 Thread Ryanne Dolan
Robin, I don't think this is an appropriate use of the user group. Ryanne On Mon, Dec 14, 2020 at 10:20 AM Robin Moffatt wrote: >  #TwelveDaysOfSMT   > > Day 5️⃣: MaskField > https://rmoff.net/2020/12/14/twelve-days-of-smt-day-5-maskfield/ > > > -- > > Robin Moffatt | Senior Developer

Re: Announcing #TwelveDaysOfSMT

2020-12-14 Thread Robin Moffatt
 #TwelveDaysOfSMT   Day 5️⃣: MaskField https://rmoff.net/2020/12/14/twelve-days-of-smt-day-5-maskfield/ -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Fri, 11 Dec 2020 at 17:14, Robin Moffatt wrote: >  #TwelveDaysOfSMT   > > Day 4️⃣: RegexRouter >