Re: Kafka Metrics in 0.8

Thanks Jun On Tue, Jul 2, 2013 at 2:35 PM, Jun Rao jun...@gmail.com wrote: In 0.8, there are more mbeans than 0.7. Thanks, Jun On Mon, Jul 1, 2013 at 9:06 PM, Hanish Bansal hanish.bansal.agar...@gmail.com wrote: Okay i'll try for same. Also want to know that in kafka-0.7 there

Re: how to keep the same partition when using mirror maker

Thanks Jun. Guodong On Wed, Jul 3, 2013 at 12:53 PM, Jun Rao jun...@gmail.com wrote: One of the issues is that in 0.7, the partitioning key is not stored on the broker. So, mirror maker won't know what key to use for partitioning. In 0.8, the partitioning key will be stored on the broker.

Re: CallbackHandler in Kafka 0.8

Hello- We receive events from external source ( for example, facebook status update events). These events are pushed to kafka queue when received There is possibility of duplicate event ( multiple facebook status update events for same account in quick intervals ) coming again and gets pushed

Re: site updates

Hi together, I have recognized following issue: https://kafka.apache.org/introduction.html in the Getting started-section the link to the design-page is broken. currently the link point to https://kafka.apache.org/08/design.html but should be https://kafka.apache.org/design.html regards

byte offset - sequential

When you guys refactored the offset to be human friendly i.e. numerical versus the byte offset, what was involved in that refactor? Is there a wiki page for that? I'm guessing there was a index file that was created for this, or is this currently managed in zookeeper? This wiki is related to

Re: Kafka Important Metrics

Good morning Joel. The bug has been filed https://issues.apache.org/jira/browse/KAFKA-958 On Mon, Jul 1, 2013 at 1:27 PM, Joel Koshy jjkosh...@gmail.com wrote: Also, there are several key metrics on the broker and client side - we should compile a list and put it on a wiki. Can you file a

Re: partition numbers and mirrormaker

Yes, but you can change num.partitions in the target brokers. Thanks, Jun On Wed, Jul 3, 2013 at 6:21 AM, Yu, Libo libo...@citi.com wrote: For a topic with N partitions on the source cluster, if I let mirrormaker to create topic automatically, the same topic on the destination cluster will

Re: byte offset - sequential

You can take a look at https://cwiki.apache.org/confluence/display/KAFKA/Keyed+Messages+Proposal Thanks, Jun On Wed, Jul 3, 2013 at 7:18 AM, S Ahmed sahmed1...@gmail.com wrote: When you guys refactored the offset to be human friendly i.e. numerical versus the byte offset, what was involved

Re: Kafka Important Metrics

I've also used jolokia, http://jolokia.org/, though it can get a little slow to respond if you don't use it right. Have rolled a JMX/HTTP 'data dumper' from scratch (can be done in a couple hundred lines of Java without too much issue)... -- Dave DeMaagd ddema...@linkedin.com | 818 262 7958

Re: Kafka Important Metrics

I haven't got the chance to try JMXTrans, although I'm planning to (and like to hear feedback if there is any). That being said they seem to focus on performance from their wiki (https://github.com/jmxtrans/jmxtrans/wiki) The JmxTransformer engine is fully multithreaded. You specify the maximum

Re: partition numbers and mirrormaker

Libo, if you set num.partitions on the target brokers and set topic auto creation to true then you don't need to create the topic beforehand. On Wed, Jul 3, 2013 at 9:15 AM, Yu, Libo libo...@citi.com wrote: Thanks, Jun. Previously I created the topic with the specified number of partitions on

Re: CallbackHandler in Kafka 0.8

Hi Nitin, We receive events from external source ( for example, facebook status update events). These events are pushed to kafka queue when received There is possibility of duplicate event ( multiple facebook status update events for same account in quick intervals ) coming again and gets

Re: site updates

Nice catch--fixed. -Jay On Wed, Jul 3, 2013 at 5:42 AM, Markus Roder roder.marku...@gmail.comwrote: Hi together, I have recognized following issue: https://kafka.apache.org/introduction.html in the Getting started-section the link to the design-page is broken. currently the link point

Re: Kafka/Hadoop consumers and producers

I guess I am more concerned about the long term than the short term. I think if you guys want to have all the Hadoop+Kafka stuff then we should move the producer there and it sounds like it would be possible to get similar functionality from the existing consumer code. I am not in a rush I just

Re: [ANNOUNCEMENT] Apache Kafka 0.8.0-beta1 Released

git pull --tags origin git tag -l Then you should see the following tag. 0.8.0-beta1 You can then checkout from the tag and see the commits. Thanks, Jun On Wed, Jul 3, 2013 at 3:02 PM, Yu, Libo libo...@citi.com wrote: Hi, I am trying to find out the main difference between the BETA1 and