Reg : Unable to produce message

All, I am getting this error for few topics WARN Error while fetching metadata with correlation id 4802 : {next=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) WARN Error while fetching metadata with correlation id 4803 : {next=LEADER_NOT_AVAILABLE}

Re: Help understanding what happened

If you hard kill the broker when it restarts it doesn't know the status of it's on-disk files, it will need to run though the last log segment to validate the checksums of messages and rebuild the index off this to ensure consistency. (Why does it need to do this validation? Because in the event

Re: Kafka Streams question

Hi Mike I'm don't have much experience with Kafka Streams yet, but from the common sense point of view, maybe it would be easier to model as Kafka Stream for actual data processing with output to another topic, that would be consumed by Kafka Connect's sinks? I see one in discussion for HBase

Not seeing disk writes on all replica brokers

Hi, I'm running version kafka_2.11-0.9.0.1 in a 2 broker configuration with 10 partitions that are replicated - replication.factor=2 Here's the output from 'kafkat partitions': TopicPartitionLeaderReplicasISRs bookmarks00[0, 1]

Re: Kafka Streams Parallelism

Hello Luis, Thank you for your detailed question! The short answer to your question: yes, Kafka Streams does apply re-partitioning for aggregations and joins, etc so that the streams can be (co-)partitioned by key. In your specific example, after the split operator, Kafka Streams will write the

Re: Larger Size Error Message

Fang, You can use the kafka.tools.DumpLogSegments to scan and view the logs, but you need the right deserializers to illustrate the content. Guozhang On Wed, Mar 16, 2016 at 4:03 PM, Fang Wong wrote: > Thanks Guozhang! > We are in the process of upgrading to 0.9.0.0. We

Re: Larger Size Error Message

Before 0.9 and before anyone knows your server host / port can produce request to you unless you have a hardware LB or firewall. In the recent release of 0.9, there is a Security feature added to Kafka, including encryption / authentication / authorization. For your case, I would suggest you

Re: Larger Size Error Message

DumpLogSegments tool is used to dump partition data logs (not application logs). Usage: ./bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files /tmp/kafka-logs/TEST-TOPIC-0/.log Use --key-decoder-class , --key-decoder-class options to pass deserializers. On Fri, Mar

Re: Kafka topics with infinite retention?

Since Kafka itself has replication, I'm not sure what HDFS backups would bring – how would you recover from e.g. all Kafka nodes blowing up if you only have an HDFS backup? Why not use MirrorMaker to replicate the cluster to a remote DC, with a process of reversing the direction in case you need

Help: Producer with SSL did not work after upgrading the kafka 0.8.2 to Kafka 0.9

Hi folks, We just finished the upgrade from 0.8.2 to 0.9 with the instructions in Kafka web site (that set the protocol version as 0.8.2.x in Kafka server 0.9). After the upgrade, we want to try the producer with SSL endpoint, but never worked. Here's the error: ~/kafka_2.11-0.9.0.0$

Re: How to assign all the partitions of a topic to consumer ?

Thanks Jason! It worked. I've missed it. On 17-Mar-2016 2:00 AM, "Jason Gustafson" wrote: > Have you looked at partitionsFor()? > > -Jason > > On Wed, Mar 16, 2016 at 4:58 AM, Kamal C wrote: > > > Hi, > > > > I'm using the new consumer in assign

KafkaStreams for counting/analytics use case

Hi everyone, Is KafkaStream the way to go to allow a light real time counting/analytics that need to do things like ? : 1) Continuously update the total count of currently open purchase orders for each user as purchase orders updates are passing thought the broker. 2) Continuously update the

Zookeeper DNS TTL

Is there any way to make Kafka 0.8.2.2 honor the DNS TTLs of its Zookeeper ensemble? I'm specifying the Zookeepers by hostname, but Kafka doesn't seem to ever re-resolve those hostnames after startup. I've tried setting the relevant JVM system properties via "-Dnetworkaddress.cache.ttl=0

Kafka Mirroring Issue: Leader Not Available Exception

I am trying to mirror a three node kafka cluster to a single node cluster. When I starts the mirror maker I got the exception LeaderNotAvailable, but at the same time topics are created and messages are commited. Please help me out. Best Regards Anshul Bhatnagar

Kafka offset moved back on rebalance

Hi, I have the 3 consumers subscribing from a single topic with the following settings. auto.commit.interval.ms = 1000 enable.auto.commit = true request.timeout.ms = 6 session.timeout.ms = 59000 heartbeat.interval.ms = 4000 Version 0.9.0. The consumers looks like this:

Re: Kafka topics with infinite retention?

> I'm not sure what HDFS backups would bring I’m not sure how realistic the threat is, but I was thinking a case in which bug in Kafka corrupts the log files. I would personally sleep better knowing there is a Kafka-independent backup of all data. > how would you recover from e.g. all Kafka

Re: Larger Size Error Message

DumpLogSegments tool is used to dump partition data logs (not application logs). Usage: ./bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files /tmp/kafka-logs/TEST-TOPIC-0/.log Use --key-decoder-class , --value-decoder-class options to pass deserializers. On Fri, Mar

Re: Kafka 0.8.2.2 is using high CPU

thanks, the new kafka cluster is stable now, I want to know the root cause. On Sat, Mar 19, 2016 at 3:34 PM, Stevo Slavić wrote: > Consider upgrading to 0.9.0.1 - our cluster is much more stable since > upgrade. Clients can be upgraded later. > > On Fri, Mar 18, 2016, 23:22

Kafka list

Hi, please add me to the Kafka list. Andreas

0.9.0.0 and 0.9.0.1 - Metadata fetching from remote server continuously fails - "Failed to find leader"

Hi, I tried 0.9.0.0 as well as 0.9.0.1 client and server combinations. Broker + Zookeeper runs on VM (Ubuntu 14.04) and consumer (console-consumer) as well as producer (Java client) runs on host (windows) Changed advertised.host.name to reflect the VM's hostname - kafka-zook. Due to firewall

Re: How to assign all the partitions of a topic to consumer ?

Have you looked at partitionsFor()? -Jason On Wed, Mar 16, 2016 at 4:58 AM, Kamal C wrote: > Hi, > > I'm using the new consumer in assign mode. I would like to assign all > the partitions of a topic to the consumer. > > For that, I need to know the number of

Re: Not seeing disk writes on all replica brokers

what exactly are you monitoring in terms of the "disk writes"? kafka writes to disk but does not fsync. so you may or may not see activity at the disk when writing; it'll be up to the OS to write back to the disk. On Wed, Mar 16, 2016 at 5:17 PM, Kiran Nagaraja wrote:

Re: Kafka offset moved back on rebalance

Hi, The offsets could belong to different partitions of the same topic. Each partition has its own offset. Try logging *record.partition()*. Regards, Daniel. On Thu, Mar 17, 2016 at 1:58 PM, Erik Pettersson wrote: > Hi, > > I have the 3 consumers subscribing from a

Re: Zookeeper DNS TTL

https://issues.apache.org/jira/browse/ZOOKEEPER-1356 which was closed as a dupe of https://issues.apache.org/jira/browse/ZOOKEEPER-338 are relevant to this... It's a zk client issue, and there are things you can do to avoid having to reconfigure the clients while you're bouncing them (CNAMEs and

Editing Kafka wiki pages (Confluence)?

Hi All, I want to publish C++ Kafka client. I have my git repository ready. Now I want add entry on Kafka “Clients” page (Confluence wiki page) for this new client. I did create my login for the Confluence and login with that. But I am not able to edit the page. Do I require to do some other

Re: Zookeeper DNS TTL

Ah I have been down this path. It is the zookeeper client. It resolves and caches the ip addresses: https://github.com/apache/zookeeper/blob/bd9a1448f9b29859092e6bdca93da121ec166b7a/src/java/main/org/apache/zookeeper/client/StaticHostProvider.java#L108 I believe they are cached forever. We have

Re: Question regarding compression of topics in Kafka

Yes it will compress the data stored on the file system if you specify compression in the producer. You can check whether the data is compressed on disk by running the following command in the data directory. kafka-run-class kafka.tools.DumpLogSegments --print-data-log --files

Re: Not seeing disk writes on all replica brokers

> > what exactly are you monitoring in terms of the "disk writes"? Looking at both bytes written and number of IOPs. I do see the batched writes activity from the OS but they seem to be missing on some replicas. I looked into the kafka log and found there is frequent change in cluster/ISR for

Re: Question regarding compression of topics in Kafka

Thanks for the response Ben. I am wondering why is "du" command not showing reduced size when compression is used. I ran an experiment with compression enabled on a topic and without compression enabled sending same amount of data in both cases. I used single node 1 replication factor Kafka

Kafka Streams scaling questions

Hi All, I read through the doc on KStreams here: http://www.confluent.io/blog/introducing-kafka-streams-stream-processing-made-simple

Kafka Streams Parallelism

Hello everyone, I've read the Kafka Streams docs available at http://docs.confluent.io/2.1.0-alpha1/streams/index.html. Since I'm coming from the world of Spark, Dataflow and friends, I couldn't avoid having some mind-breaking questions with how Kafka Streams handles its parallelism. In spark,

subscribe

subscribe

Lookup table common to all threads in a Kafka Streams app

I'm experimenting with the Kafka Streams preview and understand that joins can only happen between KStreams and/or KTables that are co-partitioned. This is a reasonable limitation necessary to support large streams. What if I have a small topic, though, that I'd like to be able to join based on