Re: Java API to monitor Consumer Offset and Lag

Otis, What's SPM that your referencing? -Steve On Fri, Mar 28, 2014 at 4:45 PM, Otis Gospodnetic < otis.gospodne...@gmail.com> wrote: > Hi Harsh, > > Are you trying to monitor lag? > SPM for Kafka has this in one of the graphs. We get this data from JMX. > There are other tools that were men

Re: Kafka and authentication

Hi Vijay, The document you pointed out has our initial thoughts on Kafka security. This work is still in design and discussion phase, no code has been written as such and we hope to pick it up in a couple months. However, if you have thoughts on how it should work and/or would like to contribute p

Re: KeeperException

I'm still confused by the messages we're getting. It seems to coincide with rebalances. Does this mean that a broker is trying to take ownership of a partition that another broker already owns? Is this a normal message that's just a side effect of a rebalance? >---

Re: Java API to monitor Consumer Offset and Lag

Hi Harsh, Are you trying to monitor lag? SPM for Kafka has this in one of the graphs. We get this data from JMX. There are other tools that were mentioned here a couple of weeks agosorry, can't recall their names now, check http://search-hadoop.com/kafka , that get this info from ZK and grap

Kafka and authentication

Hi All, I was googling around for info on securing kafka. The best document I could find was https://cwiki.apache.org/confluence/display/KAFKA/Security, which is “kind of old”. It is not clear if any steps were taken after this doc was put together. Looking at the features / bug fixes in kafka

RE: Java API to monitor Consumer Offset and Lag

I don't know if we have Java API for this. But if you dig deeper on consumer offset checker it just makes query from Zookeeper. Kafka.utils.ZkUtils does all the calls for reading data from Zookeeper. -Original Message- From: Harshvardhan Chauhan [mailto:ha...@gumgum.com] Sent: Friday,

Re: Java API to monitor Consumer Offset and Lag

I am aware of that but I should have put my question more accurately. I want to automate monitoring consumer offset and lag and wanted to know if there is a Java API for that. The ConsumerOffsetChecker has a main method and prints the information on console, I want to capture the data and push it

RE: Java API to monitor Consumer Offset and Lag

This is Scala API should be possible to call it from Groovy. -Original Message- From: Seshadri, Balaji [mailto:balaji.sesha...@dish.com] Sent: Friday, March 28, 2014 1:57 PM To: 'users@kafka.apache.org' Subject: RE: Java API to monitor Consumer Offset and Lag kafka.tools.ConsumerOffsetCh

RE: Java API to monitor Consumer Offset and Lag

kafka.tools.ConsumerOffsetChecker -Original Message- From: Harshvardhan Chauhan [mailto:ha...@gumgum.com] Sent: Friday, March 28, 2014 12:54 PM To: users@kafka.apache.org Subject: Java API to monitor Consumer Offset and Lag Hi, I am trying to write a groovy script to get consumer offset

Java API to monitor Consumer Offset and Lag

Hi, I am trying to write a groovy script to get consumer offsets and lag for our kafka cluster. Is there a Java API to get this information ? Thanks & regards, -- *Harshvardhan Chauhan* | Big Data Engineer *GumGum* | *Ads that stick* 310-260-9666 | ha...@gumgum.com

Re: Error in fetch name

Hello Tom, Yes this is normal and indicating the broker is not reachable. However, when you have a lot of topic/partitions on the brokers the log entry may be a bit verbose. We do have a switch to turn off the fetch partition info but have not enabled that. Could you file a Jira for "Enable turni

Re: data loss on replicated topic

Nope. I see lots of the "Error in fetch” messages, but that’s all. It does sound roughly like KAFKA-1193 but I don’t know how to verify if this is the same issue or not. Any ideas? On 3/27/14, 8:24 PM, "Jun Rao" wrote: >We don't expect to lose data in that case. So, this sounds like a bug. Do

Re: ZK session kill makes high level consumer replay

That makes sense. Thank you. On Thu, Mar 27, 2014 at 8:32 PM, Neha Narkhede wrote: > When I call consumer.commitOffsets(); before killing session, unit test > succeeded. This problem would happen only with autoCommit enabled > > That seems expected. If you call commitOffsets() explicitly before