Re: backup(dump) and restore environment

Hello Jakub, Maybe it will work for you to combine MirrorMaker https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330 and Burrow: https://github.com/linkedin/Burrow See recent announcement for Burrow

Re: backup(dump) and restore environment

Thanks for the quick anwser. The only think I'm not happy about is _no_ env separation. As far as I do understand, this is online operation, am I right? So the best would be to create mirror, and then cut it off form Production. greetings Jakub 2015-06-09 11:17 GMT+02:00 Stevo Slavić

Query on kafka broker and topic metadata

Hi, I am new to kafka and I have a doubt. How to read specified broker and topic statistics from kafka server? I want to read below parameters about existing topic from kafka. 1) How many activeMessages 2) How many activeSubscriptions 3) How many totalMessages 4) How many totalSubscriptions 5)

backup(dump) and restore environment

Hi I'm looking for the best way to dump current system state, and recreate it on the new, autonomic environment. Lets say I'd like to create a copy of Production, and based on that, create new, separate environment for testing. Can You suggest some solutions? greetings Jakub

Mirror maker doesn't rebalance after getting ZkNoNodeException

Hi, I have two mirror makers A and B both subscripting to the same whitelist. During topic rebalancing one of the mirror maker A encountered ZkNoNodeException and then stopped all connections. but mirror maker B didn't pick up the topics that were consumed by A and left some of the topics

Re: Query on kafka topic metadata

Hi Pavan, 1) you cannot read the current number of messages for a topic, but you can read the number of bytes for a topic in Kafka metrics. 2) you cannot read the current number of subscribers directly. However you can read from ZK and parse the results to get the subscriber counts. 3) / 4) I

Re: Mirror maker doesn't rebalance after getting ZkNoNodeException

Which version of MM are you running? On 6/9/15, 4:49 AM, tao xiao xiaotao...@gmail.com wrote: Hi, I have two mirror makers A and B both subscripting to the same whitelist. During topic rebalancing one of the mirror maker A encountered ZkNoNodeException and then stopped all connections. but

Re: offsets.storage=kafka, dual.commit.enabled=false still requires ZK

The existing consumer uses Zookeeper both to commit offsets and to assign partitions to different consumers and threads in the same consumer group. While offsets can be committed to Kafka in 0.8.2 releases, which greatly reduces the load on Zookeeper, the consumer still requires Zookeeper to

Re: offsets.storage=kafka, dual.commit.enabled=false still requires ZK

The new consumer implementation, which should be included in 0.8.3, only needs a bootstrap.servers setting and does not use a zookeeper connection. On Tue, Jun 9, 2015 at 1:26 PM, noah iamn...@gmail.com wrote: We are setting up a new Kafka project (0.8.2.1) and are trying to go straight to

Re: [ANNOUNCE] Burrow - Consumer Lag Monitoring as a Service

Hi Todd, Thanks for open sourcing this, I'm excited to take a look. It looks like it's specific to offsets stored in kafka (and not zookeeper) correct? I assume by that that LinkedIn is using the kafka storage now in production? Jason On Thu, Jun 4, 2015 at 9:43 PM, Todd Palino

producer metadata behavior when topic not created

Hi, I am talking about the 0.8.2 java producer. In our deployment, we disables auto topic creation, because we would like to control the precise number of partitions created for each topic and the placement of partitions (e.g. zone-aware). I did some experimentation and checked the code.

Stale topic for existing producer

BackgroundWe have a few consumer that *used* to consume a particular topic. This topic still exist and the consumers still exist, however, these consumer no longer subscribe to the topic. When we check the offset for the Consumer Group, kafka.tools.ConsumerOffsetChecker, ZK believes this tier

Re: [ANNOUNCE] Burrow - Consumer Lag Monitoring as a Service

For mirror maker and our audit application, we've been using Kafka-committed offsets for some time now. We've got a few other consumers who are using it, but we haven't actively worked on moving the bulk of them over. It's been less critical since we put the ZK transaction logs on SSD. And yeah,

Re: Mirror maker doesn't rebalance after getting ZkNoNodeException

I use commit 9e894aa0173b14d64a900bcf780d6b7809368384 from trunk code On Wed, 10 Jun 2015 at 01:09 Jiangjie Qin j...@linkedin.com.invalid wrote: Which version of MM are you running? On 6/9/15, 4:49 AM, tao xiao xiaotao...@gmail.com wrote: Hi, I have two mirror makers A and B both

Usage of Kafka Mirror Maker

Hi Team, I would like to know under which all circumstances I can use MirrorMaker process ? i read on a website that : *it is not really intended as a fault-tolerance mechanism* , so curious to know where all i can think of using MirrorMaker. Also, how to know the version of MirrorMaker process