Kafka Streams | Impact on rocksdb stores by Rebalancing

Hi, I would like to understand how does rebalance affect state stores migration. If I have a cluster of 3 nodes, and 1 goes down, the partitions for node3 gets assigned to node1 and node2, does the rocksdb on node1/node2 also starts updating its store from changelog topic. If yes, then what impac

Need feedbacks to consumer if hangs due to some __consumer_offsets partitions failed

hi My kafka cluster has two brokers( 0 and 1) and it created __consumer_offsets topic has 50 partitions . When broker 0 failed, half of partitions failed (because topic __consumer_offsets ReplicationFactor is 1). This results consequence that my kafkaConsumer sometimes can work w

Re: [External] Topic/Partition Assignment in streams application cluster

For what it's worth, we run 32 partitions per topic and have also observed imbalanced balancing, where a large number of A partitions are assigned to worker 1 and a large number of B partitions are assigned to worker 2, leading to imbalanced load. Nothing super bad for us yet but the effect is not

Re: Pet Project - Web based consumer for Kafka

Similar tool, but implemented very differently I believe. Full disclosure I'm not very familiar w/ Landoop's tool, but I'll try to explain the main differences from reading through that project. - Landoop's tool has a hard dependency on Confluent's Rest Proxy service, where as Kafka Web Vie

Re: Kafka Producer hangs on Init Transactions

I am getting the Co ordinator not available error when trace is enabled on kafka servers. 2018-01-08 19:57:22,088] TRACE [KafkaApi-0] Handling request:RequestHeader(apiKey=FIND_COORDINATOR, apiVersion=1, clientId=producer-2, correlationId=563) -- {coordinator_key=NBDEV1$Q4LG13020AVRO2018-01-08

Kafka Producer hangs on Init Transactions

Hi EveryOne, We are running into an issue where kafka producer hangs after inittransactions api. I have created a new topic and the same issue happens intermittently . It is not consistent but occurs more often than not . We are running confluent 4.0 and using the below dependencies on client s

Re: KSQL table retention policy

Yes, data is persistent on the brokers. And yes, you can configure those topics as you wish. For KTable, you should configure log-compaction instead of retention though. -Matthias On 1/8/18 12:00 AM, Buntu Dev wrote: > Are the tables and streams persisted on the Kafka brokers? If so then > simil

Does MirrorMaker ensures exactly-once?

Hi Everyone, since kafka 0.11.x supports exactly-once semantics, I want to be sure, that it is possible to achieve it across kafka clusters using MirrorMaker. We have got two locations with "primary" cluster in each location and for each location we have got one "aggregation" cluster which mirro

Controller connection to broker was unsuccessfull

Hi All, I am using SSL communication between the Kafka brokers. When i am sending the data, i am getting following error: WARN [Controller-3-to-broker-2-send-thread], Controller 3' connection to the broker to the broker :5010(id 2 rack:null) was unsuccessful (kafka.controller.RequestSendThread

Re: Pet Project - Web based consumer for Kafka

Is this tool similar to this: https://github.com/Landoop/kafka-topics-ui ? On Mon, Jan 8, 2018 at 7:09 AM Stephen P wrote: > Hello all! > > I've been working on a pet project over the last couple of months building > out a web based interface for consuming from Kafka called Kafka Web View >

Pet Project - Web based consumer for Kafka

Hello all! I've been working on a pet project over the last couple of months building out a web based interface for consuming from Kafka called Kafka Web View . I often found myself going to kafka-console-consumer to manually inspect or find data in

KSQL table retention policy

Are the tables and streams persisted on the Kafka brokers? If so then similar to the retention policy on the Kafka topics, is there some way to set the retention on the KSQL tables/streams based on the age of the row or something? Thanks!