Re: consumer group, why commit requests are not considered as effective heartbeats?

Hi Zaiming, Yeah, you're right. Changing coordinator won't cause a rebalance (it hasn't been that way since we added group metadata persistence). I went back and checked the code and we actually do not reset the heartbeat timer when a commit is received. I'm not sure whether there's a good reason

Re: Supervisord for Kafka 0.8.1

Thanks guys. Will try these two. On Mar 25, 2016 12:36, "Daniel Tamai" wrote: > Hi Kashyap, > > Using supervisord 3.2, this is the snippet from my supervisord.conf for > Kafka: > > [program:kafka] > command=%(ENV_KAFKA_HOME)s/bin/kafka-server-start.sh >

Bucket records based on time(kafka-hdfs-connector)

Hi kafka gurus, This might sound a little off the track, but I don't know where else to go. I tried the Confluent google group but it seems to be quite unresponsive/inactive. Please bear with me. Many thanks in advance! I am trying to copy data from Kafka into Hive tables using

Re: consumer group, why commit requests are not considered as effective heartbeats?

Hi Jason If I understand correctly, when coordinator is changed the consumer should get 'NotCoordinatorForGroup' exception not 'IllegalGenerationId'. Topic metadata change? like number of partitions changed ? I was testing it in a pretty stable cluster, and it was reproduced several times, I had

Re: How does Cloudera manager Collects Kafka Metrics

That's all information available from the jmx endpoints in Kafka. Tim On Fri, Mar 25, 2016 at 1:21 PM, yeshwanth kumar wrote: > can someone explain, how Cloudera manager Collects Kafka Metrics, such as > TotalMessages in a Topic, Total Bytes read and written from and into

How does Cloudera manager Collects Kafka Metrics

can someone explain, how Cloudera manager Collects Kafka Metrics, such as TotalMessages in a Topic, Total Bytes read and written from and into Kafka. please let me know Thanks, -Yeshwanth Can you Imagine what I would do if I could do all I can - Art of War

Re: Sessionizing inputs with Kafka Streams

Josh, If you have some ideas about improving the Transformer / Processor APIs as well as supporting sessioned windows. Please do feel free to create a JIRA and start discussions there. Also, PRs are more than welcomed :) Guozhang On Fri, Mar 25, 2016 at 10:50 AM, Guozhang Wang

Re: Sessionizing inputs with Kafka Streams

Hello Josh, We are aware that the Transformer / Processor can be improved, for example the punctuate() function should be able to return the same typed value R for Transformer. As for now, in your case you can return a sentinel from transform, and add a "filter" right after it removing sentinel

Re: Supervisord for Kafka 0.8.1

Hi Kashyap, Using supervisord 3.2, this is the snippet from my supervisord.conf for Kafka: [program:kafka] command=%(ENV_KAFKA_HOME)s/bin/kafka-server-start.sh %(ENV_CONFIG_DIR)s/server.properties stdout_logfile=%(ENV_LOG_DIR)s/kafka_out.log stderr_logfile=%(ENV_LOG_DIR)s/kafka_err.log umask=022

Re: New-Consumer group not showing up

Thanks Jason and Craig. Jason, that is exactly correct. I am using the simple consumer mode. Is there an easy way to find out the lag for consumers using the Simple Consumer mode? Regards, Ryan On Fri, Mar 25, 2016 at 11:47 AM, Jason Gustafson wrote: > Hey Ryan, > > Sounds

Re: New-Consumer group not showing up

Hey Ryan, Sounds like you might be using the so-called "simple consumer" mode. If you use assign() to give your consumer a specific partition, you're not actually using a consumer group, so there won't be any coordination with other consumers. If you use subscribe() on the other hand, then you

Re: consumer group, why commit requests are not considered as effective heartbeats?

Hi Zaiming, It rules out the most likely cause of rebalance, but not the only one. Rebalances can also be caused by a topic metadata change or a coordinator change. Can you post some logs from the consumer around the time that the unexpected rebalance occurred? -Jason On Fri, Mar 25, 2016 at

Re: Supervisord for Kafka 0.8.1

We use daemontools and this is our run file: #!/bin/bash PAC=kafka-0.8.2.x APP_HOME=/usr/share/$PAC # app options APP_CONFIG_HOME=${APP_HOME}/config APP_OPTS="${APP_CONFIG_HOME}/server.properties" JVM_OPTS="" # jvm user options if [ "abc$KAFKA_HEAP_OPTS" == "abc" ]; then export

Supervisord for Kafka 0.8.1

Hi, Am having trouble configuring Kafka server starts with supervisord. Has anyone from this group succeeded in integrating Kafka server start and stop via supervisord? Can you please share the snippet of his out of configured? Thanks Kashyap

Broker hangs all of a sudden.

Hi All, This really looks like a bug to me. I'm running Kafka 2.11-0.9.0.1. Running 2 brokers, 3 consumers. On a day to day basis, everything is working smoothly. All of a sudden, out of nowhere one of the brokers hangs and the producers lack to send messages to it, since it is the leader of some

Re: consumer group, why commit requests are not considered as effective heartbeats?

Hi Jason thanks for the reply! Forgot to mention that in we tried to test the simplest scenario in which there was only one member in the group. I think that should rule out group rebalancing right? On Thursday, March 24, 2016, Jason Gustafson wrote: > HI Zaiming, > > I