0.7 design doc?

There used to be available a very lucid page describing Kafka 0.7, its design, and the rationale behind certain decisions. I last saw it about 18 months ago. I can't find it now. Is it still available? I can find the 0.8 version, it's up there on the site. Any help? Any links? Philip

Re: If you run Kafka in AWS or Docker, how do you persist data?

Kafka on dedicated hosts running in docker under marathon under Mesos. It was a real bear to get working, but is really beautiful once I did manage to get it working. I simply run with a unique hostname constraint and number of instances = replication factor. If a broker dies and it isn't a

.deleted file descriptors

Hi, After Kafka cleaned .log / .index files based on topic retention. I can still lsof a lot of .index.deleted files. And df shows usage on disk space is accumulated to full. When this happened, just by restarting broker, it will immediately free those disk space. I seems to me kafka after

How replicas catch up the leader

Hi team, I had a replica node that was shutdown improperly due to no disk space left. I managed to clean up the disk and restarted the replica but the replica since then never caught up the leader shown below Topic:test PartitionCount:1 ReplicationFactor:3 Configs: Topic: test Partition: 0

Re: Unlimited Log Retention

Wouldn't it be a better choice to store the logs offline somewhere? HDFS and S3 are both good choices... -Mark On Feb 27, 2015, at 16:12, Warren Kiser war...@hioscar.com wrote: Does anyone know how to achieve unlimited log retention either globally or on a per topic basis? I tried

Re: cross-colo writing/reading?

Perhaps mirror maker is what you want? https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330 On Friday, February 27, 2015, Yang tedd...@gmail.com wrote: we tested our new application that reads and writes to kafka. at first we found the access latency is very high.

Re: NetworkProcessorAvgIdlePercent

Zakee, It would be useful to get the following. kafka.network:name=RequestQueueSize,type=RequestChannel kafka.network:name=RequestQueueTimeMs,request=Fetch,type=RequestMetrics kafka.network:name=RequestQueueTimeMs,request=Produce,type=RequestMetrics Thanks, Jun On Thu, Feb 26, 2015 at 2:17

Re: broker restart problems

The log seems to suggest that broker 1 is offline. Is broker 1 registered properly in ZK? You can find this out by reading the broker registration path ( https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper) from ZK. Thanks, Jun On Thu, Feb 26, 2015 at 10:31 PM,

Re: Repeated failures due to ConsumerRebalanceFailedException

Can you paste the error log for each rebalance try? You may search for keyword ³exception during rebalance². On 2/26/15, 7:41 PM, Ashwin Jayaprakash ashwin.jayaprak...@gmail.com wrote: Just give you some more debugging context, we noticed that the consumers path becomes empty after all the JVMs

Re: Kafka High Level Consumer

Great news. Thanks a lot Joe On Wed, Feb 25, 2015 at 11:46 AM, Joseph Lawson jlaw...@roomkey.com wrote: Doh that was probably my bad Pranay! A misinterpretation of some old consumer code. btw, jruby-kafka is now at 1.1.1 with proper support for deleting the offset, setting the

Re: zookeeper connection failure

Do you mean you were not able to connect to zookeeper after retry? We see this error in the log from time to time, but the zkClient will retry and usually it will succeed. Can you verify if you were finally be able to connect or not? Jiangjie (Becket) Qin On 2/27/15, 12:53 AM, Victor L

Re: Kafka metrics: high level vs simple consumer

Bok Stevo, Simple as well, if I'm not mistaken. Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/ On Fri, Feb 27, 2015 at 2:35 AM, Stevo Slavić ssla...@gmail.com wrote: Hello Apache Kafka community, In Kafka

Re: what groupID does camus use?

Thanks ! On Tue, Feb 24, 2015 at 8:23 PM, Gwen Shapira gshap...@cloudera.com wrote: Camus uses the simple consumer, which doesn't have the concept of consumer group in the API (i.e. Camus is responsible for allocating threads to partitions on its own). The client-id is hard coded and is

Re: zookeeper connection failure

I eventually figured it out: my zkClient was running on vm/guest os with zookeeper on host and vm/host port mapping was broken... On Fri, Feb 27, 2015 at 1:17 PM, Jiangjie Qin j...@linkedin.com.invalid wrote: Do you mean you were not able to connect to zookeeper after retry? We see this error

zookeeper connection failure

I seem to unable to connect single instance of Kafka server to single instance of local zookeeper server on local host (Ubuntu 14.04LTS/ open jdk 1.7.0_75): [2015-02-27 08:10:11,467] INFO Initiating client connection, connectString=localhost:2181 sessionTimeout=6000

Re: How Query Topic For Metadata

You might want ZkUtils.getPartitionsForTopic. But beware that it's an internal method that could potentially change or disappear in the future. If you're just looking for protocol-level solutions, the metadata API has a request that will return info about the number of partitions:

Unlimited Log Retention

Does anyone know how to achieve unlimited log retention either globally or on a per topic basis? I tried explicitly setting the log.retention.bytes to -1 but the default time policy kicked in after 7 days and cleaned up the messages. Thanks! Warren

Question on ISR inclusion leader election for failed replica on catchup

Hi Gang, I am testing some of the durability guarantees given by Kafka 8.2.1 which involve min in-sync replicas and disabling unclean leader election. My question is: *When will the failed replica after successfully coming up will be included back in ISR? Is this governed by

cross-colo writing/reading?

we tested our new application that reads and writes to kafka. at first we found the access latency is very high. then we realized that it's because the client and server are in different colos. moving them together reduces down the access time to 4 ms. I was wondering if there are any

Camus reads from multiple offsets in parallel?

we have a single partition, and the topic contains 300k events. we fired off a camus job, it finished within 1 minute. this is rather fast. I was guess that the multiple mappers must be reading from multiple offsets in parallel, right? otherwise if they are reading in serial (like in a consumer

RE: broker restart problems

Thanks for the reply.I confirmed that broker 1 is registered in the zk. Date: Fri, 27 Feb 2015 09:27:52 -0800 Subject: Re: broker restart problems From: j...@confluent.io To: users@kafka.apache.org The log seems to suggest that broker 1 is offline. Is broker 1 registered properly in ZK?

Re: Repeated failures due to ConsumerRebalanceFailedException

Do you see zookeeper state changed (Expired) in your logs? On Fri, Feb 27, 2015 at 10:12 AM, Jiangjie Qin j...@linkedin.com.invalid wrote: Can you paste the error log for each rebalance try? You may search for keyword ³exception during rebalance². On 2/26/15, 7:41 PM, Ashwin Jayaprakash