Re: Kafka on AWs EMR

ly 10, 2016 9:25:30 PM Subject: Re: Kafka on AWs EMR Kafka is in no way related to map reduce, this doesn't really make any sense. You could have consumers running in Hadoop listening to Kafka topics - see the Samza project. > On Jul 10, 2016, at 11:16 AM, Snehalata Nagaje > wrote: &

Kafka on AWs EMR

Hi All, I wanted to setup kafka on AWS EMR? But I could not find any documentation. Can you please help me on this? Thanks, Snehalata

Re: what is use of __consumer_offsets

: Re: what is use of __consumer_offsets Hi there, Kafka uses this topic internally for consumer offset commits. Thanks Tom Crayford Heroku Kafka On Thu, Jun 30, 2016 at 1:36 PM, Snehalata Nagaje < snehalata.nag...@harbingergroup.com> wrote: > > Hi All, > > > I am using kafka 9 version wit

what is use of __consumer_offsets

Hi All, I am using kafka 9 version with publish subscribe pattern, one consumer is listening to particular topic What is use __consumer_offsets, folders created in log files? Does it have any impact on offset commiting? Thanks, Snehalata

How to get earliest valid offset

Hi All, We are using kafka 2.10_0.9 new version, but consumer we are using old high level and low level api. I am trying to fetch earliest valid offset for topic, but it is returning latest offset if the data(log) is deleted after certain interval(which is configured in server properties)

Kafka cluster

Hi , I want to create kafka cluster for HA. Do we need to create 3 brokers?, or is it okay if create only 2 we are using only 1 partition for every topic, there is no parallelism while fetching data. Please suggest. Thanks, Snehalata

Failed to send message after three times."

Hi All, I am using kafka version kafka_2.10-0.9.0.1/ At some random time we are not able to send message to kafka It gives error as "Failed to send message after three times." is it network error? Thanks, Snehalata

Re: Kafka encryption

27;s no encryption at rest. It's recommended to use filesystem >>> encryption, or encryption of each individual message before producing it >>> for this. >>> >>> Only the new producer and consumers have SSL support. >>> >>> Thanks >>> &

Re: Kafka encryption

fore producing it > for this. > > Only the new producer and consumers have SSL support. > > Thanks > > Tom Crayford > Heroku Kafka > > On Tue, May 24, 2016 at 11:33 AM, Snehalata Nagaje < > snehalata.nag...@harbingergroup.com> wrote: > > > > &g

Re: Kafka encryption

Kumar" To: users@kafka.apache.org Sent: Tuesday, May 24, 2016 3:53:26 PM Subject: Re: Kafka encryption Yes,it does that.What specifically you are looking for? On 5/24/16, 3:52 PM, "Snehalata Nagaje" wrote: >Hi All, > > >We have requirement of encryption i

Kafka encryption

Hi All, We have requirement of encryption in kafka. As per docs, we can configure kafka with ssl, for secured communication. But does kafka also stores data in encrypted format? Thanks, Snehalata

Re: Can we delete topic in kafka

0.8.2 it works unrealiably from my experience. Can't comment on 0.9 forward. Cheers On 11.05.2016 09:48, Snehalata Nagaje wrote: > > Hi , > > Can we delete certain topic in kafka? > > I have deleted using command > > ./kafka-topics.sh --delete --topic topic_billing

Can we delete topic in kafka

Hi , Can we delete certain topic in kafka? I have deleted using command ./kafka-topics.sh --delete --topic topic_billing --zookeeper localhost:2181 It says topic marked as deletion, but it does not actually delete topic. Thanks, Snehalata

Re: Kafka 9 version offset storage mechanism changes

pic named __consumers_offsets. > > Regards, > > florin > > > > On Tue, May 10, 2016 at 8:33 AM, Gerard Klijs > > wrote: > > > >> Both are possible, but the 'new' consumer stores the offset in an > __offset > >> topic. > >> >

Kafka 9 version offset storage mechanism changes

Hi All, As per kafka 9 version, where does kafka store committed offset? is it in zookeeper or kafka broker? Also there is option to use offset storage outside kafka, does it mean , kafka will not depend on zookeepr for offset. Thanks, snehalata

How to use kafka for log aggregation

Hi All, I want to use kafka for log aggregation, how can we use it? Do we need to manually read log files and post this logs into kafka using producer API? Can Apache Flume help in it? Thanks, Snehalata

how to Update message content kafka

Hi , I want to update message content posted in kafka at particular offset? is there any API available for it? Thanks, Snehalata

Re: How to fetch old messages from kafka

e maintaining a sliding window and moving the window back only some configurable max number of times. Thanks, Mayuresh On Wed, Feb 4, 2015 at 1:29 AM, Snehalata Nagaje < snehalata.nag...@harbingergroup.com> wrote: > > > Hi Mayuresh, > > > Thanks for quick response. >

Re: How to fetch old messages from kafka

TopicA partitionA in the zookeeper. Since you are using simple consumer the offset management has to be done by your application. Thanks, Mayuresh Sent from my iPhone > On Feb 3, 2015, at 9:18 PM, Snehalata Nagaje > wrote: > > > > Hi , > > > We are using ka

How to fetch old messages from kafka

Hi , We are using kafka for storing messages in chat application. Currently we divided each topic in multiple partitions. each partition stores data for given customer who uses the application. Right now on very first request, application fetches log from kafka from earliest valid offset to

how to fetch old message from kafka

Hi , We are using kafka for storing messages in chat application. Currently we divided each topic in multiple partitions. each partition stores data for given customer who uses the application. Right now on very first request, application fetches log from kafka from earliest valid offset to

conflicted ephemeral node error

Hi , I am getting this error in kafka logs I wrote this conflicted ephemeral node [{"version":1,"brokerid":0,"timestamp":"1411969901501"}] at /controller a while back in a different session, hence I will backoff for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$) Can

Re: kafka.common.UnknownTopicOrPartitionException

ge log are automatically created if you are using kafka-0.8 version. On Wed, Oct 9, 2013 at 10:42 AM, Snehalata Nagaje < snehalata.nag...@harbingergroup.com> wrote: Actually state-change log setting is not on. So could not find the state-change log. --**-

Re: kafka.common.UnknownTopicOrPartitionException

roller and state-change log? Thanks, Jun On Tue, Oct 8, 2013 at 5:41 AM, Snehalata Nagaje < snehalata.nag...@harbingergroup.com> wrote: Hi , I am getting below exception when I consume any message from given topic, when I checked the log file,, topic is already exist in kafak server. sti

kafka.common.UnknownTopicOrPartitionException

Hi , I am getting below exception when I consume any message from given topic, when I checked the log file,, topic is already exist in kafak server. still It is giving the exceptions. kafka.common.UnknownTopicOrPartitionException: Topic topic_c7d4daa8-cdfd-40e6-b3de-a31b3ae30de9 partition 0 d

Restore zookeeper metadata

Hi , We are using central standalone zookeeper for kafka and HBase . Due to some problem, we have installed new zookeeper on different machine, but since we do not have old metadata available in Zookeeper required for kafka. We are not able to read previous topic messages. How we can restor

Re: How to get topic offset infornation

rmation to check how much your consumer is lagging or is there something else you are trying to achieve? You can use the ConsumerOffsetChecker tool to see how many messages your high level consumer is lagging behind by. Thanks, Neha On Apr 30, 2013 3:44 AM, "Snehalata Nagaje&q

How to get topic offset infornation

Hi, I am using kafka 0.8 version . I want to get topic offset information earlier and latest offset available in topic. I was able to get this using simpleconsumer API getOffseBefore. But I am using high level API to connect since I want to connect to kafka cluster using external zookeeper

RE: producer creating multiple log files after message sending

one, so that I can get all messages posted in that topic. Thanks, Snehalata -Original Message- From: Snehalata Nagaje [mailto:snehalata.nag...@harbingergroup.com] Sent: Thursday, April 18, 2013 11:41 AM To: users@kafka.apache.org Subject: RE: producer creating multiple log files after

RE: producer creating multiple log files after message sending

topic under consideration and look for "Rolling.*due to". This will tell us why the log is getting rolled over. Thanks, Neha On Wed, Apr 17, 2013 at 12:05 AM, Snehalata Nagaje wrote: > Hi, > > > > I am using kafak 0.8 version and sending messages to kafka

producer creating multiple log files after message sending

Hi, I am using kafak 0.8 version and sending messages to kafka server using below code, Producer producer = new Producer(new ProducerConfig(props)); producer.send(new KeyedMessage(topicName,partitionKey,jsonString)); producer.close(); I am able to post messages with, but after certain

How to get topic list from kafka server

Hi, I am using kafka 0.8 version. I want to know the list of topics created in kafka server along with it's metadata. Is there any API available to find out this? There is API to fetch TopicMetadata, but this needs name of topic as input parameters. I need information for all to

Kafka 0.8.0

Hello, I am using kafka 0.8.0 from git repository. And trying to post some messages, to server. It gives me error as Failed send messages after 3 tries. But when I saw, the log files, I am able to see message is posted 3 times. And also I am getting valid offset of same no of m