Kafka Consumer Behaviour

I was writing Kafka consumer and I have a query related to consumer processes. I have a consumer with groupId="testGroupId" and using the same groupId I consume from multiple topics say, "topic1" and "topic2". Also, assume "topic1" is already created on broker whereas "topic2" is not yet

Kafka Offset remains at -1

Dear Experts, I have a producer that pushes the Messages to kafka and a producer which is a spark algorithm to pick up the messages using a zookeeper instance. Below is the Spark log: Can anyone tell me what am I doing wrong? Kafka Topic details: *The command I used to create the Topic:*

Re: Kafka Offset remains at -1

Are you sure your producer and consumer are pointed to the same topic? I see RECUR_TRANS in your logs but you mentioned creating RECURRING_TRANS. If thats not the issue. I have a few questions to get more informations. - Can you list & describe the relevant topics using the kafka-topics

Re: Kafka 0.9.0 release branch

A bit off topic but does this release contain the new single threaded consumer that supports the poll interface? Thanks! On Mon, Oct 12, 2015 at 1:31 PM, Jun Rao wrote: > Hi, Everyone, > > As we are getting closer to the 0.9.0 release, we plan to cut an 0.9.0 > release

Re: Questions on the new Kafka consumer

Aah I didn't read long enough. I just read the subscribing to specific partitions, managing your own offsets and the controlling the consumer's position sections. So a follow up question - when I am subscribing to specific partitions and managing my own offsets and controlling the consumer's

Default to port 9092

Newbies may see an odd message about array index out of bounds, if they neglect to specify the : in a kafka broker list entry. In the future, could Kafka be a little smarter about this, and just assume port 9092 when one isn't explicitly provided? Other network applications tend to work this way

Questions on the new Kafka consumer

I was reading the documentation for the new Kafka consumer API at https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java and came across this: "Each Kafka consumer must specify a consumer group that it belongs to." Currently we use

Re: Partitions - Brokers - Servers

Hi, We normally run 1 broker per 1 physical server, and up to around 1000 partitions per broker (although that depends on the specific machine the broker is on and specific configuration). In order to enjoy replication, we recommend a minimum of 3 brokers in the cluster, to support 3 replicas

Re: Kafka 0.9.0 release branch

Yes, 0.9 will include the new consumer. On Tue, Oct 13, 2015 at 12:50 PM, Rajiv Kurian wrote: > A bit off topic but does this release contain the new single threaded > consumer that supports the poll interface? > > Thanks! > > On Mon, Oct 12, 2015 at 1:31 PM, Jun Rao

r/apachekafka subreddit

Any Redditors in the house? I've created an r/apachekafka subreddit for anyone interested in Reddit-based discussions. https://www.reddit.com/r/apachekafka/ -- Cheers, Andrew

partitions not syncing or becoming leaders

One of the three brokers in a Kafka cluster (v0.8.2.2) is seems be missing ISRs and even the partitions that have ISRs on it won’t be elected leaders for those partitions. Here’s a topic describe output: Topic:lia.stage.raw_events PartitionCount:20 ReplicationFactor:3

Re: r/apachekafka subreddit

Subscribed :) Since the mailing list is rather active, I'm not sure there is significant benefit in a reddit community - but I'll be around to join discussions and see how it turns out. On Tue, Oct 13, 2015 at 1:58 PM, Andrew Pennebaker < andrew.penneba...@gmail.com> wrote: > Any Redditors in

Re: [kafka-clients] Kafka 0.9.0 release branch

Not sure if I'd call it a blocker, but if we can get it in I would *really* like to see some solution to https://issues.apache.org/jira/browse/KAFKA-2397 committed. Without an explicit leave group, even normal operation of groups can leave some partitions unprocessed for 30-60s at a time under