expected exceptions?

I'm porting some unit tests from 0.7.2 to 0.8.0. The test does the following, all embedded in the same java process: -- spins up a zk instance -- spins up a kafka server using a fresh log directory -- creates a producer and sends a message -- creates a high-level consumer and verifies that it

Re: Kafka Monitoring, 0.7 vs. 0.8 JMX

0.8 JMX is different from 0.7. In 0.8, all jmx beans are exposed through metrics. One can attach a metric reporter for monitoring. Thanks, Jun On Tue, May 7, 2013 at 1:45 PM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi, We're considering adding Kafka monitoring to SPM (see

Re: Binary Data and Kafka

No. Kafka broker stores the binary data as it is. The binary data may be compressed, if compression is enabled at the producer. Thanks, Jun On Wed, May 8, 2013 at 5:57 AM, Sybrandy, Casey casey.sybra...@six3systems.com wrote: All, Does the Kafka broker Base64 encode the messages? We are

RE: Binary Data and Kafka

That's what I would have assumed. And no, we're not using compression. Thanks. From: Jun Rao [mailto:jun...@gmail.com] Sent: Wednesday, May 08, 2013 11:26 AM To: users@kafka.apache.org Cc: Sybrandy, Casey Subject: Re: Binary Data and Kafka No. Kafka broker stores the binary data as it is. The

0.8 beta release status

We are investigating a metadata related issue when there are a large number of clients (1000+). This issue, if not resolved, may cause the whole cluster to be unavailable. We are testing a fix. Once the issue is resolved, we can start the release process. Thanks, Jun

Re: Kafka Monitoring, 0.7 vs. 0.8 JMX

See http://metrics.codahale.com/getting-started/#reporting-via-http Thanks, Jun On Wed, May 8, 2013 at 9:03 AM, Dennis Haller dhal...@talenttech.comwrote: What exactly is a metric reporter - something in log4j? Thanks Dennis On Wed, May 8, 2013 at 8:20 AM, Jun Rao jun...@gmail.com

a few questions from high level consumer documentation.

Hi, I read this link https://cwiki.apache.org/KAFKA/consumer-group-example.html and have a few questions (if not too many). 1 When you say the iterator may block, do you mean hasNext() may block? 2 Remember, you can only use a single process per Consumer Group. Do you mean we can only use a

Re: a few questions from high level consumer documentation.

I'll try to answer some, the Kafka team will need to answer the others: On Wed, May 8, 2013 at 12:17 PM, Yu, Libo libo...@citi.com wrote: Hi, I read this link https://cwiki.apache.org/KAFKA/consumer-group-example.html and have a few questions (if not too many). 1 When you say the

Re: expected exceptions?

If expected, does it make sense to log them as exceptions as such? Can we instead log something meaningful to the console, like: No leader was available, one will now be created or ConsumerConnector has shutdown etc. Should I file jira's for these? Jason On Wed, May 8, 2013 at 8:22 AM,

Re: Kafka Monitoring, 0.7 vs. 0.8 JMX

Hi Jun, Does that imply that what 0.8 puts in JMX is a superset of what's in JMX in 0.7? Or have names or types of beans changed? Also, do you recommend getting metrics via JMX or via HTTP? Thanks, Otis  Monitoring for Solr / ElasticSearch / HBase / Hadoop - http://sematext.com/spm 

Re: Kafka Monitoring, 0.7 vs. 0.8 JMX

From the JmxReporter section of the metrics manual: Warning We don¹t recommend that you try to gather metrics from your production environment. JMX¹s RPC API is fragile and bonkers. For development purposes and browsing, though, it can be very useful. -Dragos On 5/8/13 2:10 PM, Otis

Re: How to create the initial zookeeper chroot path for zk.connect?

I'm seeing this issue with a single node zk instance, on my localhost. If my zkconnect is localhost:12345, it works... but if I add a chroot, e.g.: localhost:12345/xyz, I get the same error: java.lang.IllegalArgumentException: Path length must be 0 I also get the error if I do:

Re: How to create the initial zookeeper chroot path for zk.connect?

It works if I manually create the chroot first. But this is a bit cumbersome if I want to do an automated roll out to multiple deployments, etcbut workable Should I file a jira? On Wed, May 8, 2013 at 4:31 PM, Jason Rosenberg j...@squareup.com wrote: I'm seeing this issue with a

Re: a few questions from high level consumer documentation.

For #3, we need to checkpoint offsets to a central place so that if a consumer fails, another consumer in the same group can pick up from where it's left off. For #4c, leader change doesn't introduce duplicates. Thanks, Jun On Wed, May 8, 2013 at 9:17 AM, Yu, Libo libo...@citi.com wrote:

Re: expected exceptions?

Yes, could you file a jira? Please include the log messages before those exceptions. Thanks, Jun On Wed, May 8, 2013 at 9:55 AM, Jason Rosenberg j...@squareup.com wrote: If expected, does it make sense to log them as exceptions as such? Can we instead log something meaningful to the

Re: Kafka Monitoring, 0.7 vs. 0.8 JMX

Otis, 0.8 jmx is quite different from 0.7. Thanks, Jun On Wed, May 8, 2013 at 2:10 PM, Otis Gospodnetic otis_gospodne...@yahoo.com wrote: Hi Jun, Does that imply that what 0.8 puts in JMX is a superset of what's in JMX in 0.7? Or have names or types of beans changed? Also, do you

Re: How to create the initial zookeeper chroot path for zk.connect?

Actually, we already have a jira: KAFKA-294. Thanks, Jun On Wed, May 8, 2013 at 4:37 PM, Jason Rosenberg j...@squareup.com wrote: It works if I manually create the chroot first. But this is a bit cumbersome if I want to do an automated roll out to multiple deployments, etcbut

Can't connect to a server if not enough partitions

With 0.8.0, I'm seeing that an initial metadata request fails, if the number of running brokers is fewer than the configured replication factor: 877 [kafka-request-handler-0] ERROR kafka.server.KafkaApis - [KafkaApi-1946108683] Error while retrieving topic metadata

Re: expected exceptions?

Filed: https://issues.apache.org/jira/browse/KAFKA-899 https://issues.apache.org/jira/browse/KAFKA-900 Jason On Wed, May 8, 2013 at 9:22 PM, Jun Rao jun...@gmail.com wrote: Yes, could you file a jira? Please include the log messages before those exceptions. Thanks, Jun On Wed, May 8,

can the replication factor for a topic be changed after it's created?

It looks like by default, the first time a new message arrives for a given topic, it will receive the default replication factor in place on the broker at the time it is first received. Is it possible to change this later (e.g. say if we add more hardware to the cluster at a later date, etc.)?

Re: can the replication factor for a topic be changed after it's created?

If you add more brokers to an existing cluster, you can use the partition reassignment admin tool to move a replica of selected partitions over to the new broker. Thanks, Neha On Wed, May 8, 2013 at 10:27 PM, Jason Rosenberg j...@squareup.com wrote: It looks like by default, the first time a

Re: Can't connect to a server if not enough partitions

I think this error message is somewhat misleading since we create topic on the first metadata request. It is complaining that a topic with the required replication factor cannot be created if there aren't enough brokers to satisfy the replication factor. This is expected behavior whether you use