How to move Kafka data to other directories
Hi,Kafka Users, I have 3 kafka servers on the same server. I am using the /tmp/log-1,/tmp/log-2, /tmp/log-3 to place the data. Because /tmp is now full and I would move the data to other directories. I would ask what are the correct steps to do this? Is there a guide? Thanks. bit1...@163.com
Re: How to move Kafka data to other directories
I did the following steps: 1. Stop the Kafka servers 2. Copy the data from /tmp and into the new directory 3. Change the server.properties 4. Restart the Kafka servers. Looks it is now working correctly now. Not sure it is the correct way to do this, thanks! bit1...@163.com From: bit1...@163.com Date: 2015-07-06 10:30 To: users Subject: How to move Kafka data to other directories Hi,Kafka Users, I have 3 kafka servers on the same server. I am using the /tmp/log-1,/tmp/log-2, /tmp/log-3 to place the data. Because /tmp is now full and I would move the data to other directories. I would ask what are the correct steps to do this? Is there a guide? Thanks. bit1...@163.com
Re: Message loss due to zookeeper ensemble doesn't work
Can someone explain this ? Thanks! bit1...@163.com From: bit1...@163.com Date: 2015-06-25 11:57 To: users Subject: Message loss due to zookeeper ensemble doesn't work Hi, I have the offset saved in zookeeper. Because zookeeper quorum doesn't work for a short time(leader is down and new leader election).Then there is a chance that the offset doesn't write to the Zookeeper, which will lose data. I would ask whether Kafka provides some mechism for this kind of issue. bit1...@163.com
Re: Re: No key specified when sending the message to Kafka
Thank you,Ewen. I undertand it now. bit1...@163.com From: Ewen Cheslack-Postava Date: 2015-06-24 11:53 To: users@kafka.apache.org Subject: Re: No key specified when sending the message to Kafka It does balance data, but is sticky over short periods of time (for some definition of short...). See this FAQ for an explanation: https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyisdatanotevenlydistributedamongpartitionswhenapartitioningkeyisnotspecified ? This behavior has been changed in the new producer to work the way you expected, and can be overridden by providing your own Partitioner interface. On Tue, Jun 23, 2015 at 8:28 PM, bit1...@163.com bit1...@163.com wrote: I have the following code snippet that use Kafka Producer to send message(No key is specified in the KeyedMessage): val data = new KeyedMessage[String, String](topicName, msg); Kafka_Producer.send(data) Kafka_Producer is an instance of kafka.producer.Producer. With above code, I observed that the message sent to kafka is not partitioned(That is, all the messages are pushed to partition 0). If I give the message a key, they it can be partitioned across the topic. So, my question is: If no key is provided in the message , will Kafka producer not automatically partition the message with some built-in balancing algorithm? Thanks bit1...@163.com -- Thanks, Ewen
How to run Kafka in background
Hi, I am using kafak 0.8.2.1 , and when I startup Kafka with the script: ./kafka-server-start.sh ../config/server.1.properties I think it will run as background process, but when i close the terminal, the server is shutdown, which looks that it doesn't run in background Then how can I run it in background? Thanks! bit1...@163.com
Message loss due to zookeeper ensemble doesn't work
Hi, I have the offset saved in zookeeper. Because zookeeper quorum doesn't work for a short time(leader is down and new leader election).Then there is a chance that the offset doesn't write to the Zookeeper, which will lose data. I would ask whether Kafka provides some mechism for this kind of issue. bit1...@163.com
No key specified when sending the message to Kafka
I have the following code snippet that use Kafka Producer to send message(No key is specified in the KeyedMessage): val data = new KeyedMessage[String, String](topicName, msg); Kafka_Producer.send(data) Kafka_Producer is an instance of kafka.producer.Producer. With above code, I observed that the message sent to kafka is not partitioned(That is, all the messages are pushed to partition 0). If I give the message a key, they it can be partitioned across the topic. So, my question is: If no key is provided in the message , will Kafka producer not automatically partition the message with some built-in balancing algorithm? Thanks bit1...@163.com
Re: Re: Closing socket connection to /192.115.190.61. (kafka.network.Processor)
Thank you for the replay. I am using kafka_2.10-0.8.2.1,and I didn't change the log things in Kafka. bit1...@163.com From: Joe Stein Date: 2015-06-19 13:43 To: users Subject: Re: Closing socket connection to /192.115.190.61. (kafka.network.Processor) What version of Kafka are you using? This was changed to debug level in 0.8.2. ~ Joestein On Jun 18, 2015 10:39 PM, bit1...@163.com bit1...@163.com wrote: Hi, I have started the kafka server as a backgroud process, however, the following INFO log appears on the console very 10 seconds. Looks it is not an error since its log level is INFO. How could I suppress this annoying log? Thanks [2015-06-19 13:34:10,884] INFO Closing socket connection to / 192.115.190.61. (kafka.network.Processor) bit1...@163.com
Closing socket connection to /192.115.190.61. (kafka.network.Processor)
Hi, I have started the kafka server as a backgroud process, however, the following INFO log appears on the console very 10 seconds. Looks it is not an error since its log level is INFO. How could I suppress this annoying log? Thanks [2015-06-19 13:34:10,884] INFO Closing socket connection to /192.115.190.61. (kafka.network.Processor) bit1...@163.com
serveral questions about auto.offset.reset
Hi, Kafka experts: I got serveral questions about auto.offset.reset. This configuration parameter governs how consumer read the message from Kafka when there is no initial offset in ZooKeeper or if an offset is out of range. Q1. no initial offset in zookeeper means that there isn't any consumer to consume the message yet(The offset is set once the consumer starts to consume)? Q2: What does offset is out of range mean? Can you eleborate one scenario when offset is out of range could happen? auto.offset.reset has two values:smallest and largest. Assume one scenario: A producer has produced 10 messages to kafka, and there is no consumer yet to consume it. Q3: If auto.offset.reset is set to smallest, does it mean that the consumer will read the message from the offset 0?(0 is smallest here) Q4: If auto.offset.reset is set to largest, does it mean that the consumer will not read any message but wait until new messages come? bit1...@163.com
