Re: Metrics: via Broker vs. Producer vs. Consumer

Yeah totally that was what I thought you are saying. So what we have is a yammer metrics which allows you to plug in any metrics reporter (it will do jmx or whatever). But all this does it let you customize how the individual java processes output metrics. We don't solve the problem of putting all

Re: kafka 0.7 jar file

You also need to include the scala-library.jar at a minimum. If you are doing anything with ZooKeeper you will need the zookeeper and zkclient jars. Likewise, if you want Snappy compression, you will need snappy-java HTH -David On 7/24/13 4:06 PM, Nandigam, Sujitha wrote: Hi, I have one ap

kafka 0.7 jar file

Hi, I have one application based on netty client which gets messages from my server. I want to send these messages to kafka. So I wrote my producer code along with application by importing kafka jar into this project and made jar file. Now I ran application jar file to run producer class but g

Re: Added an "operations" page for 0.8 to site

Hi, Are there really a few *hundred* metrics in Kafka 0.8 vs. a few dozen(?) in Kafka 0.7? If so, I'll use something like https://github.com/sematext/jmxc to get a JMX dump to see what's in each JMX but a few hundred metrics are going to be hard to map to the bullet points under  http://ka

Re: Metrics: via Broker vs. Producer vs. Consumer

Hi, Ah, I think I didn't ask my question clearly.  Another try: * If I have a javaagent attached to the Kafka process, I'll be able to connect to its JMX and get all the Broker metrics for that Broker process. * If I have another Broker process, I'll need to attach my agent to this process, too,

Re: Meetup in Raleigh/Durham, NC

Great, thanks for presenting and sharing this David! On Wed, Jul 24, 2013 at 6:16 AM, David Arthur wrote: > Here are the slides from the meetup http://www.slideshare.net/** > mumrah/kafka-talk-tri-hug > > We had 40-50 people show up which is

Re: Metrics: via Broker vs. Producer vs. Consumer

Valid assumptions IMO Sent from my iPhone On Jul 24, 2013, at 8:22 AM, Jay Kreps wrote: > Yeah all our monitoring is just of the local process (basically just a > counter exposed through yammer metrics which support jmx and other > outputs). If I understand what you want instead of having a co

Re: invalid pom in maven central, for 0.8.0-beta1

Joe, I've verified that the version of the pom in the apache releases repo works for me (but it still has this issue: https://issues.apache.org/jira/browse/KAFKA-978). Thanks, Jason On Wed, Jul 24, 2013 at 10:37 AM, Jason Rosenberg wrote: > Joe, > > Unfortunately, I'm not sure I have an easy

Re: Metrics: via Broker vs. Producer vs. Consumer

Yeah all our monitoring is just of the local process (basically just a counter exposed through yammer metrics which support jmx and other outputs). If I understand what you want instead of having a counter that tracks, say, produce requests per second for a single broker you want one that covers th

Re: Kafka low-latency usage

Hey Jakub, Kafka has always been high throughput but has not always been low latency. In 0.7 and lower the produce request had low latency but the end-to-end latency (i.e. the time from when the message was sent to when it was delivered to the consumer) could be in the hundreds of milliseconds or

Re: Added an "operations" page for 0.8 to site

My thought was that it was kind of a pain to keep a few hundred metric names up-to-date. So we tend to end up with wrong docs. I wonder if it isn't easier just to let people fire up jconsole or whatever and browse through the full list of metrics that way? -Jay On Wed, Jul 24, 2013 at 8:06 AM, O

Re: Added an "operations" page for 0.8 to site

Hi, I know Jay recently added http://kafka.apache.org/08/ops.html - nice and useful, thanks! Would it be possible to paste all the 0.8 metrics there? Thanks, Otis -- Performance Monitoring -- http://sematext.com/spm Solr & ElasticSearch Support -- http://sematext.com/

Kafka low-latency usage

Hi all, I'm considering to use the Kafka as a low-latency message system. I want to have a system, that is able to minimise the delay between sending from producer and reading from consumer. I created a test on localhost, I created one consumer & producer and I send messages measuring the laten

Re: invalid pom in maven central, for 0.8.0-beta1

Joe, Unfortunately, I'm not sure I have an easy way to add that repo to our build system, since we use a proxy for maven resolution. I'm not seeing an easy way to add that repo, without it overriding everything in maven central (which maybe is ok, but I'm not sure). It does look like that versio

Metrics: via Broker vs. Producer vs. Consumer

Hi, I was looking at https://cwiki.apache.org/confluence/display/KAFKA/Operations#Operations-Monitoring and noticed there is no information about which metrics are available in which process/JVM/JMX. Some are available in the Broker process, but some are only available from the JVM running Consum

Re: invalid pom in maven central, for 0.8.0-beta1

For any Ivy people looking for a workaround, here is an ivysettings.xml that will resolve from Apache repos https://gist.github.com/mumrah/6070548 On 7/24/13 6:01 AM, Joe Stein wrote: Hi Jason, can you please try using https://repository.apache.org/content/repositories/releases as a repositor

Re: Meetup in Raleigh/Durham, NC

Here are the slides from the meetup http://www.slideshare.net/mumrah/kafka-talk-tri-hug We had 40-50 people show up which is about average for our meetup. Roughly half of the audience had heard of Kafka before this talk, and about 10 or so folks had used it or were using it in production. On

Re: invalid pom in maven central, for 0.8.0-beta1

Hi Jason, can you please try using https://repository.apache.org/content/repositories/releases as a repository until the beta2 release and confirm that works for you so we know the next beta will fix things before the 0.8.0 release. Thanks! /*** Joe Stein