Re: Data loss detection

Thanks, Jun. Will check and get back.. We are converting JSON to Avro and that conversion is done by the custom serializer. Our volume calculation on the producer side is based on the AVRO generic record that is passed to the producer send method— not of the encoded data output from the ser

Re: Data loss detection

Yes. We did..some output of it.. 2014-06-03 21:46:09 INFO Producer:68 - Shutting down producer 2014-06-03 21:46:09 INFO ProducerSendThread:68 - Begin shutting down ProducerSendThread 2014-06-03 21:46:09 INFO ProducerSendThread:68 - Shutdown ProducerSendThread complete 2014-06-03 21:46:09 INFO

Re: Data loss detection

By the way if you're using async producer how do you verify that you sent all the data from the producer? Do you shutdown the producer before you check? Tim On Tue, Jun 3, 2014 at 3:27 PM, Maung Than wrote: > Thanks, Tim. > > We are just trying to benchmark the kafka producers and there is no i

Re: Data loss detection

We have a metric on msg rate in both the producer and the broker. Could you see if they match? Thanks, Jun On Tue, Jun 3, 2014 at 2:13 PM, Maung Than wrote: > Hi, > > We are seeing less data on the brokers than we send form the producers: > 84 GB to 58 GB. > > What is the best way to ensure

Re: topics load balancing within a consumer group

Currently, we distribute partitions to consumers on a per topic basis. So, in your cases, consumer 1 will get all the data. Thanks, Jun On Tue, Jun 3, 2014 at 3:11 PM, Weide Zhang wrote: > Hi, > > I have a question regarding load balancing within a consumer group. > > Say I have a consumer gr

Re: Data loss detection

Thanks, Tim. We are just trying to benchmark the kafka producers and there is no issue with cluster or brokers being down in this case. We are seeing way less data on the borers after calculating the sizes of the logs on the brokers) and there is no compression. We send 84 GB, but total log

Re: [DISCUSS] Kafka Security Specific Features

I think that¹s one option. What I would offer here is that we need to separate out the concepts of authorization and authentication. Authentication should definitely be modular, so that we can plug in appropriate schemes depending on the organization. For example, you may want client certificates,

topics load balancing within a consumer group

Hi, I have a question regarding load balancing within a consumer group. Say I have a consumer group of 4 consumers which subscribe to 4 topics , each of which have one partition. Will there be rebalancing happening on topic level ? Or I will expect consumer 1 have all the data ? Weide

Re: Data loss detection

Hi Maung, If your required.acks is 1 then the producer only ensures that one broker receives the data before it's sucessfully returned to the client. Therefore if the broker crashes and lost all the data then you lose data, or similarly it can happen even before the data is fsynced. To ensure th

Data loss detection

Hi, We are seeing less data on the brokers than we send form the producers: 84 GB to 58 GB. What is the best way to ensure / detect if all data has been send properly to the brokers from the producers. Is there any logs that we can check on the producers? Configuration is 5 Brokers, 2 pr

Re: event-shuttle

This does come up often, interesting solution. And another reason to dig more into Go so I can give this a spin. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop

event-shuttle

Thought Id post this here, as Ive seen questions here around "What do I do if kafka is down?" here is one possibility. https://github.com/sclasen/event-shuttle this is a go service meant to run as a unix daemon, and apps running on the same box can just post messages over http to a service on lo

Re: [DISCUSS] Kafka Security Specific Features

... client specific presented information, signed in some way, listing topic permissions. read, write, list. TLS lends itself to client certificates. On Jun 3, 2014, at 12:57 PM, Joe Stein wrote: > 4) Authorization > > We should have a policy of "404" for data, topics, partitions (etc) if >

[DISCUSS] Kafka Security Specific Features

Hi,I wanted to re-ignite the discussion around Apache Kafka Security. This is a huge bottleneck (non-starter in some cases) for a lot of organizations (due to regulatory, compliance and other requirements). Below are my suggestions for specific changes in Kafka to accommodate security requirements

Re: kafka-reassign-partitions.sh leaves data on source broker?

Json file should be correct, and I have just checked the code, it should send the stop replica request to the old leaders to delete their logs. If you can consistently re-produce this issue could you file a jira? Guozhang On Sat, May 31, 2014 at 7:40 PM, Otis Gospodnetic < otis.gospodne...@gmail

Re: How to delete Topic in kafka 0.8.1 ?

For now deleting topics would be a bit tedious and you need to stop the brokers during the deletion. Here is what you can do: 1. Stop the brokers. 2. Delete the partition folders for this topic on all brokers. 3. Delete the topic-partition znode in ZK under the /brokers path. 4. Restart the broker

Re: How to delete Topic in kafka 0.8.1 ?

There is probably some steps, but unfortunately I'm not too sure about them so I prefer to kept my mouth shut :) But from what I read in the link I sent you, even if it will be only available in the next release, there is a patch available and the code has been commit to trunk https://issues.apach

Re: Kafka Broker frequent shutdown with exception

Hi Robert, I tried *kill -SIGTERM * comand to stop Kafka Broker but it shutdown all other brokers as well. Can you please suggest me, How can I avoid stopping other brokers ? Is there any configuration changes required ? Regards, Rafeeq S On Tue, Jun 3, 2014 at 9:40 AM, Robert Hodges w

Re: How to delete Topic in kafka 0.8.1 ?

Thanks François, Then how we can delete topics in kafka 0.8.1 ? Is there any procedure/steps to delete topic ? Thanks for your kind response! On Tue, Jun 3, 2014 at 3:57 PM, François Langelier wrote: > The delete topic isn't working ATM > > I think it will be available in the next release >

Re: How to delete Topic in kafka 0.8.1 ?

The delete topic isn't working ATM I think it will be available in the next release https://issues.apache.org/jira/browse/KAFKA-1397 François Langelier Étudiant en génie Logiciel - École de Technologie Supérieure Capitaine Club Capra VP-Communi

Re: How to delete Topic ?

The delete topic isn't working ATM I think it will be available in the next release https://issues.apache.org/jira/browse/KAFKA-1397 François Langelier Étudiant en génie Logiciel - École de Technologie Supérieure Capitaine Club Capra VP-Communic

How to delete Topic in kafka 0.8.1 ?

I am using Kafka 0.8.1 and I tried to execute Topic delete command but still Topic list showing that specific Topic. Topic Delete commad which i used: *bin/kafka-topics.sh --zookeeper zk_host:port/chroot --delete --topic my_topic_name* How can i verify that my_topic is deleted ? Is there specif

How to delete Topic ?

I am using Kafka 0.8.1 and I tried to execute Topic delete command but still Topic list showing that specific Topic. Topic Delete commad which i used: *bin/kafka-topics.sh --zookeeper zk_host:port/chroot --delete --topic my_topic_name* How can i verify that my_topic is deleted ? Is there specif

Re: Kafka Broker frequent shutdown with exception

Thanks, Using 'kill -SIGTERM ', can assure *No data loss* ? if yes then my problem will resolve. Thanks for your kind response! On Tue, Jun 3, 2014 at 9:40 AM, Robert Hodges wrote: > Hi Rafeeq, > > You can stop them individually by killing the processes. The > kafka-server-stop.sh comma