Re: Java API to monitor Consumer Offset and Lag

Hi Harsh, Are you trying to monitor lag? SPM for Kafka has this in one of the graphs. We get this data from JMX. There are other tools that were mentioned here a couple of weeks agosorry, can't recall their names now, check http://search-hadoop.com/kafka , that get this info from ZK and grap

Re: Hardware planning

Ray, We are, for SPM . On c1.medium instances, I believe, we have: * Jetty receiving tens of thousands of metrics per second (in batches, so the rate of HTTP requests is lower than that number_ * Kafka brokers * ZK instances So far we have not had issues with this. Knock

Re: 0.8.1 stability

Hi, We are in the process of migrating to 0.8.1, so we haven't seen any bugs yet, but would welcome a bugfix release if others are reporting issues. Versioning Q: wouldn't that be 0.8.2? Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://semate

Re: Large number of sockets to JMX port in CLOSE_WAIT on 0.7.2 broker

I don't think this is a Kafka problem. We read from Kafka JMX every few seconds with SPM and don't see this problem with 0.7.2. Maybe a Jolokia bug? Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Wed, Mar 12, 2014 at 4:0

Re: [ANNOUNCEMENT] Apache Kafka 0.8.1 Released

Yes: https://cwiki.apache.org/confluence/display/KAFKA/Migrating+from+0.7+to+0.8 Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Wed, Mar 12, 2014 at 6:01 PM, Kane Kane wrote: > Is it possible to update from 0.8 on the f

Re: memory requirement for kafka - how to estimate

Hi, I'm not sure if anyone can give you exact numbers, but I think you'll find Kafka brokers very light. We run them on I think medium EC2 instances and Kafka brokers they put hardly any strain on them. If it helps give you an idea, go to https://apps.sematext.com/demo and look for "SA.Prod

Re: Anouncing Kafka Offset Monitor 0.1

offsets, you would probably > not have to change anything else in the code. > > P > > > On Sat, Mar 8, 2014 at 2:48 AM, Otis Gospodnetic < > otis.gospodne...@gmail.com > > wrote: > > > Hi, > > > > Does this work with Kafka 0.7.x or does Kafka 0.7.x n

Re: Anouncing Kafka Offset Monitor 0.1

Hi, Does this work with Kafka 0.7.x or does Kafka 0.7.x not expose the info needed for computing the lag? Thanks, Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Fri, Mar 7, 2014 at 1:49 PM, Pierre Andrews wrote: > Hello

Re: It would be nice to be able to read Kafka topics in reverse.

Hi, We have a similar use case. In our case it's about performance metrics and situations where we plan catch up. Instead of processing data chronologically, it would make sense to process newest data first, so performance metrics graphs for "now" have data. Here is a related thread: http://sea

Re: C++ Producer => Broker => Java Consumer?

m > > > > > > On Fri, Jan 31, 2014 at 2:36 PM, Philip O'Toole > wrote: > > > > > Is this a Kafka C++ lib you wrote yourself, or some open-source > library? > > > What version of Kafka? > > > > > > Philip > > > > >

C++ Producer => Broker => Java Consumer?

Hi, If Kafka Producer is using a C++ Kafka lib to produce messages, how can Kafka Consumers written in Java deserialize them? Thanks, Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/

0.8.1 ETA?

Hi, I hate asking the When question, but I was under the impression 0.8.1 was around the corner so I looked at JIRA and found 90 opened issues labeled as 0.8.1. https://issues.apache.org/jira/issues/?jql=project%20%3D%20KAFKA%20AND%20fixVersion%20%3D%20%220.8.1%22%20AND%20resolution%20%3D%20U

Re: Calcuate Consumer lag from JMX beans in Kafka 0.7.2

Hi Xuyen, SPM for Kafka gets all this stuff already. I didn't look into whether/how exactly one can spot consumer lag, but if you spot a way to do it, please share. There is a demo of SPM for Kafka if you go to https://apps.sematext.com/demoso you can see all Kafka performance graphs and see if

Re: Kafka Authenication and Monitoring Tools

Hi Joseph, For the monitoring piece there is SPM - see http://blog.sematext.com/2013/10/16/announcement-spm-performance-monitoring-for-kafka/. It will monitor your ZooKeeper, too. Kafka uses ZK, so you may want to keep an eye on it as well. Otis -- Performance Monitoring * Log Analytics * Searc

Re: count of bytes stored in kafka

Hi, Not a direct way to see this, but we use Kafka 0.7.2 and monitor the disk space on it, so that's one way to see *roughly* what is happening. If you use something like SPM to monitor your Kafka nodes (see http://blog.sematext.com/2013/10/16/announcement-spm-performance-monitoring-for-kafka/) an

Re: Consuming "backwards"?

Hi, On Fri, Dec 6, 2013 at 6:32 PM, Steven Parkes wrote: > On Dec 6, 2013, at 2:03 PM, Otis Gospodnetic > wrote: > > > but I think the > > problem is that each time we grab we could get some of the same messages > we > > already processed > > Doesn't

Re: Consuming "backwards"?

realized that wouldn't work actually (starting at the > > end > > > >> is > > > >>>> fine)... the fetch size being taken in is still going to increment > > > >> forward > > > >>>> ... > > > >>>> > > >

Re: Consuming "backwards"?

> >>>> > >>>> so let me take that all back and say... can't be done now but I think > it > >>>> is feasible to be done with some broker modifications to read the log > >>>> differently... off the top of my head can't think of

Re: Consuming "backwards"?

csearch Support * http://sematext.com/ > On Friday, December 6, 2013, Otis Gospodnetic wrote: > > > Hi, > > > > Does Kafka offer a way to consume messages in batches, but "from the > end"? > > > > This would be valuable to have in all systems where t

Consuming "backwards"?

Hi, Does Kafka offer a way to consume messages in batches, but "from the end"? This would be valuable to have in all systems where the most recent data is a lot more important than older data, such as performance metrics, and maybe even logsmaybe also trading/financial data, and such. Any ch

Re: kafka producer - retry messages

t; >> > >> I may be missing something, but I never fully understand why people try > >> really hard to build a stream-to-disk backup approach, when they might > be > >> able to couple tightly to Kafka, which, well, just streams to disk. > >> > >> P

Re: kafka producer - retry messages

Hi, We've done this at Sematext, where we use Kafka in all 3 products/services you see in my signature. When we fail to push a message into Kafka we store it in the FS and from there we can process it later. Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch S

[ANNOUNCEMENT] Kafka Performance Monitoring in SPM

Hi, Just a quick mention here that we've recently added support for monitoring Kafka to SPM. This is in addition to monitoring Hadoop, HBase, SenseiDB, Solr, Elasticseach, JVM, OS metrics, custom metrics, etc. See: http://blog.sematext.com/2013/10/16/announcement-spm-performance-monitoring-for-ka

Re: Metrics: via Broker vs. Producer vs. Consumer

e patience and the info, Jay! Otis Performance Monitoring for Solr / ElasticSearch / HBase / Hadoop - http://sematext.com/spm  > > From: Jay Kreps >To: "users@kafka.apache.org" ; Otis Gospodnetic > >Sent: Thursday, July 25, 2013

Re: Added an "operations" page for 0.8 to site

ole or whatever and browse >through the full list of metrics that way? > >-Jay > > >On Wed, Jul 24, 2013 at 8:06 AM, Otis Gospodnetic < >otis.gospodne...@gmail.com> wrote: > >> Hi, >> >> I know Jay recently added http://kafka.apache.org/08/ops.html -

Re: Metrics: via Broker vs. Producer vs. Consumer

that. > >I'm not sure how valid any of those assumptions actually are. > >-Jay > > >On Wed, Jul 24, 2013 at 7:29 AM, Otis Gospodnetic < >otis.gospodne...@gmail.com> wrote: > >> Hi, >> >> I was looking at >> >> https://cwiki.apache.org/

Re: Added an "operations" page for 0.8 to site

Hi, I know Jay recently added http://kafka.apache.org/08/ops.html - nice and useful, thanks! Would it be possible to paste all the 0.8 metrics there? Thanks, Otis -- Performance Monitoring -- http://sematext.com/spm Solr & ElasticSearch Support -- http://sematext.com/

Metrics: via Broker vs. Producer vs. Consumer

Hi, I was looking at https://cwiki.apache.org/confluence/display/KAFKA/Operations#Operations-Monitoring and noticed there is no information about which metrics are available in which process/JVM/JMX. Some are available in the Broker process, but some are only available from the JVM running Consum

Re: Kafka Monitoring, 0.7 vs. 0.8 JMX

rics. One can attach a metric reporter for monitoring. > >Thanks, > >Jun > > >On Tue, May 7, 2013 at 1:45 PM, Otis Gospodnetic > wrote: > >> Hi, >> >> We're considering adding Kafka monitoring to SPM (see >> http://sematext.com/spm/index.html ). 

Kafka Monitoring, 0.7 vs. 0.8 JMX

Hi, We're considering adding Kafka monitoring to SPM (see http://sematext.com/spm/index.html ). We use Kafka ourselves and would like to see our Kafka metrics in SPM along with Hadoop, HBase, Solr, and other metrics we monitor. My questions: * What do people currently use for monitoring Kafka? *

Re: Analysis of producer performance

Hi, Is there anything one can do to "defend" from: "Trying to push more data than the brokers can handle for any sustained period of time has catastrophic consequences, regardless of what timeout settings are used. In our use case this means that we need to either ensure we have spare capacity

Kafka's archives on search-hadoop.com (and new Kafka website?)

Hi, Some time ago people on this list asked if we could add Kafka to search-hadoop.com and we just did that yesterday. You can see Kafka and all other projects there if you go to http://search-hadoop.com/ Kafka-only stream is at http://search-hadoop.com/Kafka I say Jay emailed about the new web

Re: Upgrade from 0.8 trunk to 0.8 GA?

her few weeks before deploying 0.8 in production, if you'd like to >avoid incompatible releases. However, upgrading from 0.7.2 to 0.8 is >also backwards incompatible and will require careful deployment >planning to correctly release 0.8 without service interruptions. > >HTH,

<    1   2