Re: Differences between new and legacy scala producer API

2015-05-08 Thread Manoj Khangaonkar
On Thu, May 7, 2015 at 10:01 PM, Rendy Bambang Junior rendy.b.jun...@gmail.com wrote: Hi - Legacy scala api for producer is having keyed message along with topic, key, partkey, and message. Meanwhile new api has no partkey. Whats the difference between key and partkey? In the new API, key

Re: Kafka consumer offset checker hangs indefinitely

2015-05-08 Thread Mayuresh Gharat
Is X.X.X.X:9092 up and accepting connections? I am confused aas in why is it not connecting some other broker if connection to this broker fails. Can you check if the broker is up? The way it works is the consumer will send a ConsumerMetadataRequest to one of the brokers and get the offsetmanager

Re: Kafka consumer offset checker hangs indefinitely

2015-05-08 Thread Meghana Narasimhan
Hi Mayuresh, Yes, the broker is up and accepting connections. Multiple consumers are consuming off topics on the broker. Also I am seeing the issue only with this particular version (2.11-0.8.2.1). It worked fine with the beta that I was using earlier. On Fri, May 8, 2015 at 12:45 PM, Mayuresh

Re: New Java producer broker metadata update stuck

2015-05-08 Thread Mayuresh Gharat
Also it would be great to know if you see the same issue when you don't have different ip - broker id mapping. Also it would be great if you can explain different ip - broker id mapping mean as Becket said. Thanks, Mayuresh On Fri, May 8, 2015 at 9:48 AM, Mayuresh Gharat

Fwd: Kafka consumer offset checker hangs indefinitely

2015-05-08 Thread Meghana Narasimhan
Hi, I'm using the Kafka 8.2.1 version(kafka_2.11-0.8.2.1) and the consumer offset checker hangs indefinitely and does not return any results. I enabled the debug for tools and below is the debug statements as seen on the stdout. Any thoughts or inputs on this will be much appreciated. command

New Java producer broker metadata update stuck

2015-05-08 Thread Dan
Hi, We've noticed an issue on our staging environment where all 3 of our Kafka hosts shutdown and came back with a different ip - broker id mapping. I know this is not good and we're fixing that separately. But what we noticed is all the consumers recovered but the producers got stuck with the

Re: New Java producer broker metadata update stuck

2015-05-08 Thread Jiangjie Qin
Dan, Just to make sure I understand it correctly. What do you mean by different ip - broker mapping? Do you mean you changed your broker ip? We have different mechanism in and producer to get the cluster information. Consumer get all the information from Zookeeper while producer has to talk to

Re: New Java producer broker metadata update stuck

2015-05-08 Thread Mayuresh Gharat
It should do a updateMetadataRequest in case it gets NOT_LEADER_FOR PARTITION. This looks like a bug. Thanks, Mayuresh On Fri, May 8, 2015 at 8:53 AM, Dan danharve...@gmail.com wrote: Hi, We've noticed an issue on our staging environment where all 3 of our Kafka hosts shutdown and came

RE: Support https or ssl

2015-05-08 Thread Jamie Wang
Thank you Harsha for the reply. Will the patch work with 0.8.2.1? So the general logic is that we write an agent that sends message via https and then may be a servlet using Kafka produce class to authenticate with kafka server on ssl port and so on When will the full implementation be

Is there a way to know when I've reached the end of a partition (consumed all messages) when using the high-level consumer?

2015-05-08 Thread James Cheng
Hi, I want to use the high level consumer to read all partitions for a topic, and know when I have reached the end. I know the end might be a little vague, since items keep showing up, but I'm trying to get as close as possible. I know that more messages might show up later, but I want to know

RE: Support https or ssl

2015-05-08 Thread Sriharsha Chintalapani
Jamie,       Once the patch gets in you can configure kafka brokers with ssl configs and similarly you can configure producer and consumer with ssl both will negotiate a ssl connection once its established rest of the communication will happen over ssl. I am not sure about the timeline but we

Issue with kafka-topics.sh tool for adding new partitions with replica assignment

2015-05-08 Thread Stefan Schadwinkel
Hi, with Kafka 0.8 it is possible to add new partitions on newly added brokers and supply a partition assignment to put the new partitions mainly on the new brokers (4 and 5 are the new brokers): bin/kafka-add-partitions.sh --topic scale-test-001 \ --partition 14 \ --replica-assignment-list

Kafka consumer offset checker hangs indefinitely

2015-05-08 Thread Meghana Narasimhan
Hi, I'm using the Kafka 8.2.1 version(kafka_2.11-0.8.2.1) and the consumer offset checker hangs indefinitely and does not return any results. I enabled the debug for tools and below is the debug statements as seen on the stdout. Any thoughts or inputs on this will be much appreciated. command