Re: Consumer Offsets and Open FDs

We ran into this as well, and I ended up with the following that works for us. log.cleaner.dedupe.buffer.size=536870912 log.cleaner.io.buffer.size=2000 On 13/07/2016 14:01, "Lawrence Weikum" wrote: >Apologies. Here is the full trace from a broker: > >[2016-06-24

Re: Kafka without Storm/Spark

Definitely. You can read off kafka using the samples shown in KafkaConsumer javadoc, transform if necessary and publish to the destination topic. On 01/07/2016 03:24, "numan goceri" wrote: >Hello everyone, >I've a quick question:I'm using Apache Kafka

Re: How to work around log compaction error (0.8.2.2)

kbase.com/t/kafka/users/159jbe18en/log-cleaner-thread-stops > > >On Thu, Apr 28, 2016 at 1:44 AM, Rakesh Vidyadharan < >rvidyadha...@gracenote.com> wrote: > >> Hello, >> >> We enabled log compaction on a few topics, as we want to preserve >> permanently

How to work around log compaction error (0.8.2.2)

Hello, We enabled log compaction on a few topics, as we want to preserve permanently the latest versions of messages published to specific topics. After enabling compaction, the log cleaner thread dies with the same error for the topics we tried it on. It looks like kafka has starting offset

Re: Log cleaner error

Found the issue. My publisher was not assigning a key to all messages. On 07/03/2016 14:40, "Rakesh Vidyadharan" <rvidyadha...@gracenote.com> wrote: >Hello, > >We are using Kafka 0.8.2.2 and have modified most of our topics to use log >compaction and a shorter r

Log cleaner error

Hello, We are using Kafka 0.8.2.2 and have modified most of our topics to use log compaction and a shorter retention.ms equivalent to 24 hours for those topics. We did not see any change in the disk usage, and on looking at the log cleaner log file, noticed the following message (repeated on

Re: Migrating to clustered environment

Found the issue. Our cluster is on AWS, and the third node had not set the advertised.host.name property which is required on AWS. Set that and replication completed successfully. On 05/02/2016 10:17, "Rakesh Vidyadharan" <rvidyadha...@gracenote.com> wrote: >Hello, >

Unable to set log.cleanup.policy on topic

Hello, I upgraded to the latest 0.8 release 0.8.2.2, and tried to set the log cleanup policy on some of our topics (I got same error with 0.8.1, but thought it may have been some issue with that release). /opt/kafka/bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic

Re: Unable to set log.cleanup.policy on topic

configuration property (see >http://kafka.apache.org/documentation.html#topic-config ). Since you are >configuring particular topic, you need to use second one. > >Kind regards, >Stevo Slavic. > >On Mon, Dec 7, 2015 at 8:59 PM, Rakesh Vidyadharan < >rvidyadha...@gracenote.com

Failed attempt to delete topic

Hello, We are on an older kafka (0.8.1) version. While a number of consumers were running, we attempted to delete a few topics using the kafka-topics.sh file (basically want to remove all messages in that topic and restart, since our entities went through some incompatible changes). We

Re: Failed attempt to delete topic

) Thanks Rakesh On 03/12/2015 17:33, "Mayuresh Gharat" <gharatmayures...@gmail.com> wrote: >Can you paste some logs from the controller, when you deleted the topic? > >Thanks, > >Mayuresh > >On Thu, Dec 3, 2015 at 2:30 PM, Rakesh Vidyadharan < >rv

Re: Failed attempt to delete topic

Mayuresh > >On Thu, Dec 3, 2015 at 4:04 PM, Rakesh Vidyadharan < >rvidyadha...@gracenote.com> wrote: > >> Thanks Stevo. I did see some messages related to /admin/delete_topics. >> Will do some research on how I can clean up zookeeper. >> >> Thanks >&g

Re: Failed attempt to delete topic

; wrote: > >> Can you paste some logs from the controller, when you deleted the topic? >> >> Thanks, >> >> Mayuresh >> >> On Thu, Dec 3, 2015 at 2:30 PM, Rakesh Vidyadharan < >> rvidyadha...@gracenote.com> wrote: >> >> > Hello, >>

Re: Avro messages from Kafka topic

We publish messages to kafka in Thrift format. We use the old simple consumer and just retrieve the message bytes, transform back to object model using Thrift API and do whatever our application needs with it. On 20/10/2015 11:08, "Buntu Dev" wrote: >I got a Kafka topic

Re: Launch Kafka/Zookeeper servers without using command line

Hello, I tried using the sources for unit tests, but have been unsuccessful in getting it to work with the low level API (kafka version 0.8.2.2). Our code is modelled on the Simple Consumer API code, and the error always occurs in the findLeader code. SEVERE: Error communicating with

replication factor: 1 larger than available brokers: 0 issue

Hello, We have been using Kafka 0.8.1 for a few months and this morning we started seeing a lot of these types of errors. We started trying to publish messages to a few additional topics, but topic creation (automatic creation mode) always fails with the following type of error: