Re: 0.8.2 How do I suppress INFO Closing socket connection to
Oh, great, I get it. Thank you. On Mon, Jan 4, 2016 at 6:26 PM, tao xiao <xiaotao...@gmail.com> wrote: > You can bump the log level to warn for a particular class > > log4j.logger.kafka.network.Processor=WARN > > On Tue, 5 Jan 2016 at 08:33 Dillian Murphey <crackshotm...@gmail.com> > wrote: > > > Constant spam of this INFO on my log. > > > > [2016-01-05 00:31:15,887] INFO Closing socket connection to /10.9.255.67 > . > > (kafka.network.Processor) > > [2016-01-05 00:31:15,917] INFO Closing socket connection to /10.9.255.67 > . > > (kafka.network.Proces > > > > Anyone know what I need to modify (assuming log4j.properties) to disable > > this but still be informed of issues? > > > > Thank you! > > >
Re: Frequent ZK session timeouts
[2016-01-12 <http://airmail.calendar/2016-01-12%2012:00:00%20PST> 22:16:59,629 <http://airmail.calendar/2016-01-12%2022:16:59%20PST>] TRACE [Controller 925537]: leader imbalance ratio for broker 925537 is 0.00 (kafka.controller.KafkaController) [2016-01-12 <http://airmail.calendar/2016-01-12%2012:00:00%20PST> 22:21:07,167 <http://airmail.calendar/2016-01-12%2022:21:07%20PST>] INFO [SessionExpirationListener on 925537], ZK expired; shut down all controller components and try to re-elect (kafka.controller.KafkaController$SessionExpirationListener) [2016-01-12 <http://airmail.calendar/2016-01-12%2012:00:00%20PST> 22:21:07,167 <http://airmail.calendar/2016-01-12%2022:21:07%20PST>] INFO [delete-topics-thread-925537], Shutting down (kafka.controller.TopicDeletionManager$DeleteTopicsThread) [2016-01-12 <http://airmail.calendar/2016-01-12%2012:00:00%20PST> 22:21:07,169 <http://airmail.calendar/2016-01-12%2022:21:07%20PST>] INFO [delete-topics-thread-925537], Shutdown completed (kafka.controller.TopicDeletionManager$DeleteTopicsThread) [2016-01-12 <http://airmail.calendar/2016-01-12%2012:00:00%20PST> 22:21:07,169 <http://airmail.calendar/2016-01-12%2022:21:07%20PST>] INFO [delete-topics-thread-925537], Stopped (kafka.controller.TopicDeletionManager$Del This occurs very frequently, even after clean slating kafka. This is something that never occurs in our production env. I've read here and there that it could be a GC issue? Here is the tail end of recent GC log. 20534K(8354560K), 52.5293140 secs] [Times: user=209.09 sys=0.06, real=52.53 secs] 2016-01-11T23:16:05.149+: 784.219: [GC 784.219: [ParNew: 274263K->1685K(306688K), 54.8993730 secs] 793174K->520803K(8354560K), 54.8994450 secs] [Times: user=218.86 sys=0.03, real=54.90 secs] 2016-01-11T23:17:01.095+: 840.165: [GC 840.165: [ParNew: 274325K->1896K(306688K), 56.4208930 secs] 793443K->521139K(8354560K), 56.4209750 secs] [Times: user=224.88 sys=0.05, real=56.42 secs] 2016-01-11T23:17:59.024+: 898.093: [GC 898.093: [ParNew: 274536K->1705K(306688K), 58.1100630 secs] 793779K->521093K(8354560K), 58.1101400 secs] [Times: user=231.75 sys=0.05, real=58.12 secs] 2016-01-11T23:18:58.240+: 957.310: [GC 957.310: [ParNew: 274345K->1483K(306688K), 64.2820420 secs] 793733K->521047K(8354560K), 64.2821180 secs] [Times: user=241.93 sys=0.06, real=64.28 secs] 2016-01-11T23:20:03.571+: 1022.640: [GC 1022.640: [ParNew: 274123K->1379K(306688K), 61.5305280 secs] 793687K->521097K(8354560K), 61.5305990 secs] [Times: user=245.72 sys=0.01, real=61.53 secs] 2016-01-11T23:21:06.194+: 1085.263: [GC 1085.263: [ParNew: 274019K->1508K(306688K), 63.4433440 secs] 793737K->521372K(8354560K), 63.4434240 secs] [Times: user=253.33 sys=0.02, real=63.44 secs] 2016-01-11T23:22:10.413+: 1149.482: [GC 1149.483: [ParNew: 274148K->1313K(306688K), 65.6956010 secs] 794012K->521330K(8354560K), 65.6956660 secs] [Times: user=262.01 sys=0.05, real=65.69 secs] Heap par new generation total 306688K, used 132112K [0x0005f5a0, 0x00060a6c, 0x00060a6c) eden space 272640K, 47% used [0x0005f5a0, 0x0005fd9bbba0, 0x00060644) from space 34048K, 3% used [0x00060644, 0x0006065884a8, 0x00060858) to space 34048K, 0% used [0x00060858, 0x00060858, 0x00060a6c) concurrent mark-sweep generation total 8047872K, used 520016K [0x00060a6c, 0x0007f5a0, 0x0007f5a0) concurrent-mark-sweep perm gen total 38760K, used 25768K [0x0007f5a0, 0x0007f7fda000, 0x0008) On Tue, Jan 12, 2016 at 6:34 PM, Mayuresh Gharat <gharatmayures...@gmail.com > wrote: > Can you paste the logs? > > Thanks, > > Mayuresh > > On Tue, Jan 12, 2016 at 4:58 PM, Dillian Murphey <crackshotm...@gmail.com> > wrote: > > > Possibly running more stable with 1.7 JVM. > > > > Can someone explain the Zookeeper session? SHould it never expire, > unless > > the broker becomes unresponsive? I set a massive timeout value in the > > broker config far beyond the amount of time I see the zk expiration. Is > > this entirely on the kafka side, or could zookeeper be doing something? > > From my zk logs I didn't see anything unusual, just exceptions as a > result > > of the zk session expiring (my guess). > > > > tnx > > > > On Tue, Jan 12, 2016 at 3:05 PM, Dillian Murphey < > crackshotm...@gmail.com> > > wrote: > > > > > Our 2 node kafka cluster has become unhealthy. We're running zookeeper > > as > > > a 3 node system, which very light load. > > > > > > What seems to be happening is in the controller log we get a ZK session > > > expire message, and in the process of re-assigning the lead
Frequent ZK session timeouts
Our 2 node kafka cluster has become unhealthy. We're running zookeeper as a 3 node system, which very light load. What seems to be happening is in the controller log we get a ZK session expire message, and in the process of re-assigning the leader for the partitions (if I'm understanding this right, please correct me), the broker goes offline and it interrupts our applications that are publishing messages. We don't see this in production, and kafka has been stable for months, since september. I've searched a lot and found some similiar complaints but no real solutions. I'm running 0.8.2 and JVM 1.6.X on ubuntu. Thanks for any ideas at all.
0.8.2 How do I suppress INFO Closing socket connection to
Constant spam of this INFO on my log. [2016-01-05 00:31:15,887] INFO Closing socket connection to /10.9.255.67. (kafka.network.Processor) [2016-01-05 00:31:15,917] INFO Closing socket connection to /10.9.255.67. (kafka.network.Proces Anyone know what I need to modify (assuming log4j.properties) to disable this but still be informed of issues? Thank you!
Find current kafka memory usage
I was running out of heap space for my kafka broker. Is there a way I can check how much memory kafka is using so I can alert myself if it is reaching the max heap size? Default is 1GB. Thanks
Re: Increasing replication factor reliable?
Alright, thank you all. Appreciate it. Cheers On Wed, Nov 25, 2015 at 10:50 PM, Gaurav Agarwal <gaurav130...@gmail.com> wrote: > So u have two nodes running where you want to increase the replication > factor 2 because of fault tolerance. That won't be a problem > On Nov 25, 2015 6:26 AM, "Dillian Murphey" <crackshotm...@gmail.com> > wrote: > > > Is it safe to run this on an active production topic? A topic was > created > > without a replication factor of 2 and I want to increase it from 1 to 2 > to > > have fault tolerance. > > > > > > > http://kafka.apache.org/documentation.html#basic_ops_increase_replication_factor > > >
Re: Is re-partition hitless process?
Not adding. Taking some of the partitions from one kafka server and spreading them to another. On Mon, Nov 23, 2015 at 5:40 PM, Gwen Shapira <g...@confluent.io> wrote: > By re-partition you mean adding partitions to an existing topics? > > There are two things to note in that case: > 1. It is "hitless" because all it does is create new partitions where > future records can go, it does not actually move data around. > 2. You could be "hit" if your consumer code assumes that all data for a > specific key will be in the same partition. Because all future keys may get > assigned to different partitions than what they had before adding > partitions. > > Hope this helps. > > Gwen > > On Mon, Nov 23, 2015 at 9:38 AM, Dillian Murphey <crackshotm...@gmail.com> > wrote: > > > Can I do this on a production system and not have downtime? I'm using > > kafkamanager to make this easier, but it's just running the re-partition > > task. > > >
Re: Is re-partition hitless process?
I see, thanks. Yes, re-assignment. Got my terminology off. On Tue, Nov 24, 2015 at 4:45 PM, Gwen Shapira <g...@confluent.io> wrote: > ah, re-assignment! > > If your re-assignment involves moving leaders to other servers, there will > be a tiny downtime when the new leader election happens. > Otherwise the main risk is copying the partition over to the new node - it > can use lots of network and IO. On a busy system we recommend scripting > this to do very few (maybe one) partition at a time. > > On Tue, Nov 24, 2015 at 4:42 PM, Dillian Murphey <crackshotm...@gmail.com> > wrote: > > > Not adding. Taking some of the partitions from one kafka server and > > spreading them to another. > > > > On Mon, Nov 23, 2015 at 5:40 PM, Gwen Shapira <g...@confluent.io> wrote: > > > > > By re-partition you mean adding partitions to an existing topics? > > > > > > There are two things to note in that case: > > > 1. It is "hitless" because all it does is create new partitions where > > > future records can go, it does not actually move data around. > > > 2. You could be "hit" if your consumer code assumes that all data for a > > > specific key will be in the same partition. Because all future keys may > > get > > > assigned to different partitions than what they had before adding > > > partitions. > > > > > > Hope this helps. > > > > > > Gwen > > > > > > On Mon, Nov 23, 2015 at 9:38 AM, Dillian Murphey < > > crackshotm...@gmail.com> > > > wrote: > > > > > > > Can I do this on a production system and not have downtime? I'm > using > > > > kafkamanager to make this easier, but it's just running the > > re-partition > > > > task. > > > > > > > > > >
Increasing replication factor reliable?
Is it safe to run this on an active production topic? A topic was created without a replication factor of 2 and I want to increase it from 1 to 2 to have fault tolerance. http://kafka.apache.org/documentation.html#basic_ops_increase_replication_factor
Is re-partition hitless process?
Can I do this on a production system and not have downtime? I'm using kafkamanager to make this easier, but it's just running the re-partition task.
Question on re-partition
I want change the partition assignment to spread the partitions across two machines, since machine #1 is getting full on disk space. I have kafka manager to make this easy. Is there any downtime to re-assigning partitions? I assume kafka builds up the new partitions and then does a hit-less switch over. Thanks for any info.
Occasional Time Outs - GC problem or...?
Just poking around our amazon kafka instance. Occasionally one of our applications gets a kafka time-out. We want to guess it is a momentary network issue with aws, however, I noticed the GC log is updated very frequently. Could the GC be causing a problem? How do I determine if it is causing a problem which could lead to a temporary socket failure to kafka? Thanks for any ideas.
Re: Keeping Zookeeper and Kafka Server Up
supervisord is pretty easy to use. Netflix Exhibitor will manage this all for zookeeper, if you want to try that tool. On Wed, Jun 17, 2015 at 7:03 AM, Kashyap Mhaisekar kashya...@gmail.com wrote: We use supervisord for this. It ensures that the processes are always up and running. Thanks Kashyap On Wednesday, June 17, 2015, Shayne S shaynest...@gmail.com wrote: kafka-server-start.sh has a -daemon option, but I don't think Zookeeper has it. On Tue, Jun 16, 2015 at 11:32 PM, Su She suhsheka...@gmail.com javascript:; wrote: It seems like nohup has solved this issue, even when the putty window becomes inactive the processes are still running (I din't need to interact with them). I might look into using screen or tmux as a long term solution. Thanks Terry and Mike! Best, Su On Tue, Jun 16, 2015 at 3:42 PM, Terry Bates terryjba...@gmail.com javascript:; wrote: Greetings, nohup does the trick, as Mr. Bridge has shared. If you seem to want to run these and still have some interactivity with the services, consider using screen or tmux as these will enable you to run these programs in foreground, have added windows you can use to access shell, tail logs, and so on, and enable you to disconnect from the session, but still have these sessions available for re-attachment. In addition, I using runit for service supervision may enable you to keep daemons running, but if your services are dying you may need to introspect more deeply on the root cause versus working around it by restarting them. *Terry Bates* *Email: *terryjba...@gmail.com javascript:; *Phone: (*412) 215-0881 *Skype*: terryjbates *GitHub*: https://github.com/terryjbates *Linkedin*: http://www.linkedin.com/in/terryjbates/ On Tue, Jun 16, 2015 at 3:30 PM, Mike Bridge m...@bridgecanada.com javascript:; wrote: Have you tried using nohup nohup bin/zookeeper-server-start.sh config/zookeeper.properties nohup bin/kafka-server-start.sh config/server.properties On Tue, Jun 16, 2015 at 3:21 PM, Su She suhsheka...@gmail.com javascript:; wrote: Hello Everyone, I'm wondering how to keep Zookeeper and Kafka Server up even when my SSH (using putty) becomes inactive. I've tried running it in the background (using ), but it seems like it stops sometimes after a couple hours or so and I'll have to restart zookeeper and/or the kafka server. The only remediation i've found is to export TMOUT=[big number], but there must be another solution. Thank you! Best, Su
Re: Increased replication factor. Replication didn't happen!
Thank you Adam, that helps me know what to look for.
On Thu, Jun 11, 2015 at 12:49 PM, Adam Dubiel dubiel.a...@gmail.com wrote:
I just tried it out on my 0.8.2 cluster and it worked just fine - the ISR
grew, replica factor changed and data was physically moved to new brokers.
Was there not output/no logs? I see things like
INFO Created log for partition [topicName,7] in /opt/kafka/ with
properties {.. some json}
in server.log on new brokers.
2015-06-11 21:02 GMT+02:00 Dillian Murphey crackshotm...@gmail.com:
Oh, hmm. There is even documentation on it:
http://kafka.apache.org/documentation.html#basic_ops_increase_replication_factor
Having a difficult time sifting through the logs. Is this not a common
operation? Do users normally just delete the topic and create a new
one??
On Wed, Jun 10, 2015 at 8:20 PM, Shady Xu shad...@gmail.com wrote:
Right now, Kafka topics do not support changing replication factor or
partition number after creation. The kafka-reassign-partitions.sh tool
can
only reassign existent partitions.
2015-06-11 9:31 GMT+08:00 Gwen Shapira gshap...@cloudera.com:
What do the logs show?
On Wed, Jun 10, 2015 at 5:07 PM, Dillian Murphey
crackshotm...@gmail.com wrote:
Ran this:
$KAFKA_HOME/bin/kafka-reassign-partitions.sh
But Kafka did not actually do the replication. Topic description
shows
the
right numbers, but it just didn't replicate.
What's wrong, and how do I trigger the replication to occur??
I'm running 0.8.2.0
thanks
Noobie question about consumers and ordering of messages
So I just realized that when creating multiple partitions for a topic, it is the responsibility of the producer and the consumer to decide where to put data. So if I am a consumer, how do I know what partition to read data from? What if the producer only put data in 1 partition out of 50. And so there isn't any order, here, right? Only order per partition, but there is no concept of consuming messages in the order they were produced, correct? Thanks. Just struggling on the practical aspects now that I've moved past some of the administration tasks.
Re: Increased replication factor. Replication didn't happen!
Oh, hmm. There is even documentation on it: http://kafka.apache.org/documentation.html#basic_ops_increase_replication_factor Having a difficult time sifting through the logs. Is this not a common operation? Do users normally just delete the topic and create a new one?? On Wed, Jun 10, 2015 at 8:20 PM, Shady Xu shad...@gmail.com wrote: Right now, Kafka topics do not support changing replication factor or partition number after creation. The kafka-reassign-partitions.sh tool can only reassign existent partitions. 2015-06-11 9:31 GMT+08:00 Gwen Shapira gshap...@cloudera.com: What do the logs show? On Wed, Jun 10, 2015 at 5:07 PM, Dillian Murphey crackshotm...@gmail.com wrote: Ran this: $KAFKA_HOME/bin/kafka-reassign-partitions.sh But Kafka did not actually do the replication. Topic description shows the right numbers, but it just didn't replicate. What's wrong, and how do I trigger the replication to occur?? I'm running 0.8.2.0 thanks
Increased replication factor. Replication didn't happen!
Ran this: $KAFKA_HOME/bin/kafka-reassign-partitions.sh But Kafka did not actually do the replication. Topic description shows the right numbers, but it just didn't replicate. What's wrong, and how do I trigger the replication to occur?? I'm running 0.8.2.0 thanks
Re: Waiting for replication factor to take effect...?
Nothing on this? Someone must surely have a need to add brokers and increase the replication factor. I waited a day and I still do not see in-sync replicas. :( On Wed, May 27, 2015 at 5:51 PM, Dillian Murphey crackshotm...@gmail.com wrote: Hi Ho, I'm trying to increase my replication factor from 1 to 2. I used the tool kafka-reassign-partitions.sh I see the replica factor change, but my replicas are not syncing up. There is pretty much no data in this topic. Will this replication get triggered at some point?? root@b3b4b5d71b48:/opt/kafka_2.10-0.8.2.1# $KAFKA_HOME/bin/kafka-topics.sh --describe --zookeeper $ZK --topic mytopic Topic:mytopic PartitionCount:2 ReplicationFactor:2 Configs: Topic: mytopic Partition: 0 Leader: 925537 Replicas: 9255216,925537 Isr: 925537 Topic: mytopic Partition: 1 Leader: 9255216 Replicas: 925537,9255216 Isr: 9255216 Thanks for any comments.
Waiting for replication factor to take effect...?
Hi Ho, I'm trying to increase my replication factor from 1 to 2. I used the tool kafka-reassign-partitions.sh I see the replica factor change, but my replicas are not syncing up. There is pretty much no data in this topic. Will this replication get triggered at some point?? root@b3b4b5d71b48:/opt/kafka_2.10-0.8.2.1# $KAFKA_HOME/bin/kafka-topics.sh --describe --zookeeper $ZK --topic mytopic Topic:mytopic PartitionCount:2 ReplicationFactor:2 Configs: Topic: mytopic Partition: 0 Leader: 925537 Replicas: 9255216,925537 Isr: 925537 Topic: mytopic Partition: 1 Leader: 9255216 Replicas: 925537,9255216 Isr: 9255216 Thanks for any comments.
Delete topic pending
If a broker doesn't have the topic, and I run delete topic, that topic will be in a pending delete state forever. What am I doing wrong here? Also, what if I have data loss and I just want to delete the dang topic form zookeeper directly with non of this pending stuff. Thanks
Re: Kafka behind AWS ELB
I'm interested in this topic as well. If you put kafka brokers inside an autoscaling group, then AWS will automatically add brokers if demand increases, and the ELB will automatically round-robin across all of your kafka instances. So in your config files and code, you only need to provide a single DNS name (the load balancer). You don't need to specify all your kafka brokers inside your config file. If a broker dies, the ELB will only route to healthy nodes. So you get a lot of robustness, scalability, and fault-tolerance by using the AWS services. Kafka Brokers will automatically load balance, but the question is whether it is ok to put all your brokers behind an ELB and expect the system to work properly. What alternatives are there to dynamic/scalable broker clusters? I don't want to have to modify my config files or code if I add more brokers, and I want to be able to handle a broker going down. So these are the reasons AWS questions like this come up. Thanks for any comments too. :) On Mon, May 4, 2015 at 9:03 AM, Mayuresh Gharat gharatmayures...@gmail.com wrote: Ok. You can deploy kafka in AWS. You can have brokers on AWS servers. Kafka is not a push system. So you will need someone writing to kafka and consuming from kafka. It will work. My suggestion will be to try it out on a smaller instance in AWS and see the effects. As I do not know the actual use case about why you want to use kafka for, I cannot comment on whether it will work for you personalized use case. Thanks, Mayuresh On Mon, May 4, 2015 at 8:55 AM, Chandrashekhar Kotekar shekhar.kote...@gmail.com wrote: I am sorry but I cannot reveal those details due to confidentiality issues. I hope you understand. Regards, Chandrash3khar Kotekar Mobile - +91 8600011455 On Mon, May 4, 2015 at 9:18 PM, Mayuresh Gharat gharatmayures...@gmail.com wrote: Hi Chandrashekar, Can you please elaborate the use case for Kafka here, like how you are planning to use it. Thanks, Mayuresh On Sat, May 2, 2015 at 9:08 PM, Chandrashekhar Kotekar shekhar.kote...@gmail.com wrote: Hi, I am new to Apache Kafka. I have played with it on my laptop. I want to use Kafka in AWS. Currently we have tomcat web servers based REST API. We want to replace REST API with Apache Kafka, web servers are behind ELB. I would like to know if we can keep Kafka brokers behind ELB? Will it work? Regards, Chandrash3khar Kotekar Mobile - +91 8600011455 -- -Regards, Mayuresh R. Gharat (862) 250-7125 -- -Regards, Mayuresh R. Gharat (862) 250-7125
Re: Kafka brokers behind AWS ELB
Can you clarify what is meant by using the ELB as a metadata broker? Let's say I have a python script that sends messages into kafka. Do I have to provide a list of brokers or can I use the ELB DNS alias that will round-robin to all my healthy brokers? Much easier to use the ELB instead of hard-coding server names or IPs. And when the broker list expands or shrinks, I have to them edit my python code? Thanks for any help. On Sun, May 3, 2015 at 11:05 AM, Jiangjie Qin j...@linkedin.com.invalid wrote: Yes. On 5/3/15, 10:57 AM, Chandrashekhar Kotekar shekhar.kote...@gmail.com wrote: Thanks a lot for the information. So basically each broker needs to have his own public IP address? Regards, Chandrash3khar Kotekar Mobile - +91 8600011455 On Sun, May 3, 2015 at 11:09 PM, Jiangjie Qin j...@linkedin.com.invalid wrote: You can use ELB address as metadata broker list. But you still need to allow direct traffic from clients to a brokers because clients need to talk to a specific broker who has the interested partition. Jiangjie (Becket) Qin On 5/2/15, 11:10 AM, Chandrashekhar Kotekar shekhar.kote...@gmail.com wrote: Hi, I am new to Apache Kafka. I have played with it on my laptop. I want to use Kafka in AWS. Currently we have tomcat web servers based REST API. We want to replace REST API with Apache Kafka, web servers are behind ELB. I would like to know if we can keep Kafka brokers behind ELB? Will it work? Regards, Chandrash3khar Kotekar Mobile - +91 8600011455
Re: Topic missing Leader and Isr
Thank you, Buntu.
On Sat, May 2, 2015 at 1:16 PM, Buntu Dev buntu...@gmail.com wrote:
This seems to be an issue thats fixed in 0.8.2 and after upgrading the
kafka cluster to 0.8.2 this issue was resolved.
https://issues.apache.org/jira/browse/KAFKA-1738
https://issues.apache.org/jira/browse/KAFKA-1738
On Fri, May 1, 2015 at 2:58 PM, Dillian Murphey crackshotm...@gmail.com
wrote:
Hey, not a help, but I just posted the same question. I'm noticing the
same thing. Where is your hardware? Just curious. I'm running my
kafka/zookeeper on AWS.
On Tue, Apr 28, 2015 at 10:23 AM, Buntu Dev buntu...@gmail.com wrote:
Also note that the metadata for the topic is missing. I tried creating
few
more topics and all have the same issue.
Using the Kafka console producer on the topic, I see these error
messages
indicating the missing metadata:
WARN Error while fetching metadata [{TopicMetadata for topic my-topic
-
No partition metadata for topic my-topic due to
kafka.common.UnknownException}] for topic [my-topic]: class
kafka.common.UnknownException (kafka.producer.BrokerPartitionInfo)
Re: Topic missing Leader and Isr
Oh let me further add. I believe there is no leader/isr when I've
re-created a deleted topic. If I pick a topic name never seen before I get
a leader/isr. Were you deleting topics? Deleting topics is also a good
test of your system. FYI:
$KAFKA_HOME/bin/kafka-run-class.sh kafka.admin.DeleteTopicCommand
--zookeeper $ZK --topic test
On Fri, May 1, 2015 at 2:58 PM, Dillian Murphey crackshotm...@gmail.com
wrote:
Hey, not a help, but I just posted the same question. I'm noticing the
same thing. Where is your hardware? Just curious. I'm running my
kafka/zookeeper on AWS.
On Tue, Apr 28, 2015 at 10:23 AM, Buntu Dev buntu...@gmail.com wrote:
Also note that the metadata for the topic is missing. I tried creating few
more topics and all have the same issue.
Using the Kafka console producer on the topic, I see these error messages
indicating the missing metadata:
WARN Error while fetching metadata [{TopicMetadata for topic my-topic -
No partition metadata for topic my-topic due to
kafka.common.UnknownException}] for topic [my-topic]: class
kafka.common.UnknownException (kafka.producer.BrokerPartitionInfo)
Re: Topic missing Leader and Isr
Hey, not a help, but I just posted the same question. I'm noticing the
same thing. Where is your hardware? Just curious. I'm running my
kafka/zookeeper on AWS.
On Tue, Apr 28, 2015 at 10:23 AM, Buntu Dev buntu...@gmail.com wrote:
Also note that the metadata for the topic is missing. I tried creating few
more topics and all have the same issue.
Using the Kafka console producer on the topic, I see these error messages
indicating the missing metadata:
WARN Error while fetching metadata [{TopicMetadata for topic my-topic -
No partition metadata for topic my-topic due to
kafka.common.UnknownException}] for topic [my-topic]: class
kafka.common.UnknownException (kafka.producer.BrokerPartitionInfo)
Re: Leaderless topics
kafka_2.9.2-0.8.1.1 Thanks, Gwen. On Thu, Apr 30, 2015 at 5:34 PM, Gwen Shapira gshap...@cloudera.com wrote: Which Kafka version are you using? On Thu, Apr 30, 2015 at 4:11 PM, Dillian Murphey crackshotm...@gmail.com wrote: Scenerio with 1 node broker, and 3 node zookeeper ensemble. 1) Create topic 2) Delete topic 3) Re-create with same name I'm noticing this recreation gives me Leader: non, and Isr: as empty. Any ideas what the deal is here? I googled around and not being an experienced kafka admin, someone said to delete the /controller entry in zk. This appears to fix the problem on existing topics that show no leader. Is it ok to do this? What am I doing by deleting /controller? Is there a better way? Thanks for any advice, and your time of course.
Sometimes I don't get a leader with 1 broker
Running a 1 broker system. I had some issues with the system but got it working. I've deleted the topic I had trouble with and re-created it. But describing shows no leader, not producer/consumption works on it. I create a brand new topic with a name I never used before and I get a leader. I think I sometimes get a leader and sometimes don't. Not sure. controller log is fine. Is this normal?
Kafka still aware of old zookeeper nodes
I had 3 zookeeper nodes. I added 3 new ones and shut down the old 3. The server.log shows Closing socket connection error to the old IPs. I rebooted the kafka server entirely but it still somehow seems aware of these servers. Any ideas what's up?
Delete topic / Recreate = No leader
I am trying to reproduce this. But if I create a topic, then delete it, then re-create it, no leader is getting assigned. I can still produce/consume messages (via command line, basic testing). Is there some additional cleanup I need to do? Thanks for your time!
Re: Kafka still aware of old zookeeper nodes
Not sure if this is the best way to do this, but my zookeeper.connect is set to a DNS alias which points to a load balancer for 3 zookeeper nodes. I was trying this to see if I could have the kafka config dynamic and allow me to change/scale whatever I wanted with zookeeper and not have to ever mess with the config for kafka. Thanks for your comments. On Thu, Apr 30, 2015 at 11:35 AM, svante karlsson s...@csi.se wrote: Have you changed zookeeper.connect= in server.properties. A better procedure for replacing zookeeper nodes would be to shutdown one and install the new one with the same ip. This can easily be done to a running cluster. /svante 2015-04-30 20:08 GMT+02:00 Dillian Murphey crackshotm...@gmail.com: I had 3 zookeeper nodes. I added 3 new ones and shut down the old 3. The server.log shows Closing socket connection error to the old IPs. I rebooted the kafka server entirely but it still somehow seems aware of these servers. Any ideas what's up?
Leaderless topics
Scenerio with 1 node broker, and 3 node zookeeper ensemble. 1) Create topic 2) Delete topic 3) Re-create with same name I'm noticing this recreation gives me Leader: non, and Isr: as empty. Any ideas what the deal is here? I googled around and not being an experienced kafka admin, someone said to delete the /controller entry in zk. This appears to fix the problem on existing topics that show no leader. Is it ok to do this? What am I doing by deleting /controller? Is there a better way? Thanks for any advice, and your time of course.
Re: Poor performance running performance test
You could be right Ewen. I was starting to wonder about the load balancer too. Is using a load balancer a bad idea? How else do users know which kafka broker to connect to? I'm using one of the IPs directly and I don't see that error. I am seeing an occasional connection refused. What the heck. Maybe this is another aws specific thing. OR, I am running kafka brokers in a docker container. I think I will remove the docker component and see if that makes a difference. Thanks for the reply. bow
question on the mailing list
Hi all, Sorry for asking, but is there some easier way to use the mailing list? Maybe a tool which makes reading and replying to messages more like google groups? I like the hadoop searcher, but the UI on that is really bad. tnx
Re: Consuming Kafka Messages Inside of EC2 Instances
Am I understanding your question correctly... You're asking how do you establish connectivity to an instance in a private subnet from the outside world? Are you thinking in terms of zookeeper or just general aws network connectivity? On Wed, Jan 28, 2015 at 11:03 AM, Su She suhsheka...@gmail.com wrote: Hello All, I have set up a cluster of EC2 instances using this method: http://blogs.aws.amazon.com/bigdata/post/Tx2D0J7QOVRJBRX/Deploying-Cloudera-s-Enterprise-Data-Hub-on-AWS As you can see the instances are w/in a private subnet. I was wondering if anyone has any advice on how I can set up a Kafka zookeeper/server on an instance that receives messages from a Kafka Producer outside of the private subnet. I have tried using the cluster launcher, but I feel like it is not a best practice and only a temporary situation. Thank you for the help! Best, Su
WARN Error in I/O with NetworkReceive.readFrom(NetworkReceive.java
Running the performance test. What is the nature of this error?? I'm running a very high end cluster on aws. Tried this even within the same subnet on aws. bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance topic9 5000 100 -1 acks=1 bootstrap.servers=$IP:9092 buffer.memory=67108864 batch.size=8196 2015-01-28 16:32:22,178] WARN Error in I/O with /IP ADDR (org.apache.kafka.common.network.Selector) java.io.EOFException at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:62) at org.apache.kafka.common.network.Selector.poll(Selector.java:248) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:192) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122) at java.lang.Thread.run(Thread.java:745) 65910 records sent, 13158.3 records/sec (1.25 MB/sec), 38525.9 ms avg latency, 39478.0 max latency. Thanks for any ideas
Poor performance running performance test
I'm a new user/admin to kafka. I'm running a 3 node ZK and a 6 brokers on aws. The performance I'm seeing is shockingly bad. I need some advice! bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance test2 5000 100 -1 acks=1 bootstrap.servers=5myloadbalancer:9092 buffer.memory=67108864 batch.size=8196 6097 records sent, 13198.3 records/sec (1.26 MB/sec), 2098.0 ms avg latency, 4306.0 max latency. 71695 records sent, 14339.0 records/sec (1.37 MB/sec), 6658.1 ms avg latency, 9053.0 max latency. 65195 records sent, 13028.6 records/sec (1.24 MB/sec), 11504.0 ms avg latency, 13809.0 max latency. 71955 records sent, 14391.0 records/sec (1.37 MB/sec), 16137.4 ms avg latency, 18541.0 max latency. Thanks for any help!
Re: Poor performance running performance test
I was running the performance command from a virtual box server, so that seems like it was part of the problem. I'm getting better results running this on a server on aws, but that's kind of expected. Can you look at these results, and comment on the occasional warning I see? I appreciate it! 1220375 records sent, 243928.6 records/sec (23.26 MB/sec), 2111.5 ms avg latency, 4435.0 max latency. 1195090 records sent, 239018.0 records/sec (22.79 MB/sec), 2203.1 ms avg latency, 4595.0 max latency. 1257165 records sent, 251433.0 records/sec (23.98 MB/sec), 2172.6 ms avg latency, 4525.0 max latency. 1230981 records sent, 246196.2 records/sec (23.48 MB/sec), 2173.5 ms avg latency, 4465.0 max latency. [2015-01-28 07:19:07,274] WARN Error in I/O with myawsloadbalancer(org.apache.kafka.common.network.Selector) java.io.EOFException at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:62) at org.apache.kafka.common.network.Selector.poll(Selector.java:248) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:192) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122) at java.lang.Thread.run(Thread.java:745) 1090689 records sent, 218137.8 records/sec (20.80 MB/sec), 2413.6 ms avg latency, 4829.0 max latency. On Tue, Jan 27, 2015 at 7:37 PM, Ewen Cheslack-Postava e...@confluent.io wrote: Where are you running ProducerPerformance in relation to ZK and the Kafka brokers? You should definitely see much higher performance than this. A couple of other things I can think of that might be going wrong: Are all your VMs in the same AZ? Are you storing Kafka data in EBS or local ephemeral storage? If EBS, have you provisioned enough IOPS. On Tue, Jan 27, 2015 at 4:29 PM, Dillian Murphey crackshotm...@gmail.com wrote: I'm a new user/admin to kafka. I'm running a 3 node ZK and a 6 brokers on aws. The performance I'm seeing is shockingly bad. I need some advice! bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance test2 5000 100 -1 acks=1 bootstrap.servers=5myloadbalancer:9092 buffer.memory=67108864 batch.size=8196 6097 records sent, 13198.3 records/sec (1.26 MB/sec), 2098.0 ms avg latency, 4306.0 max latency. 71695 records sent, 14339.0 records/sec (1.37 MB/sec), 6658.1 ms avg latency, 9053.0 max latency. 65195 records sent, 13028.6 records/sec (1.24 MB/sec), 11504.0 ms avg latency, 13809.0 max latency. 71955 records sent, 14391.0 records/sec (1.37 MB/sec), 16137.4 ms avg latency, 18541.0 max latency. Thanks for any help! -- Thanks, Ewen
Command to list my brokers
Hi all, I just want a way to query all of my brokers to see if they're all connected and online, without creating a topic. Or is creating a topic the best way to verify all my brokers are up and running??? Thanks
kafka cluster on aws
I can't seem to find much information to help me (being green to kafka) on setting up a cluster on aws. Does anyone have any sources? The question I have off the bat is, what methods have already been explored to generate a unique broker id? If I spin up a new server, do I just need to maintain my own broker-id list somewhere so I don't re-use an already allocated broker id? Also, I read an article about a broker going down and requiring a new broker be spun up with the same id. Is this also something I need to implement? I want to setup a kafka auto-scaling group on AWS, so I can add brokers at well or based on load. It doesn't seem too complicated, or maybe I'm too green to see it, but I don't want to re-invent everything myself. I know Loggly uses AWS/Kafka, so I'm hunting for more details on that too. Thanks for any help
Re: kafka cluster on aws
Thanks for the comments. Hey Joe, I'm looking at your project now. I'm going to give it a try.
Re: kafka cluster on aws
Trying to understand the docs. Can I just use the docker image and run the minotaur command from there? I don't understand the Basion SSH stuff. Do I need that? I just want a quick start for right now. Also, not sure where I get the ENVIRONMENT.key. Any extra help is greatly appreciated. You can email directly. Thanks! On Wed, Jan 14, 2015 at 12:09 PM, Joe Stein joe.st...@stealth.ly wrote: We have an open source framework you can use to spin up Kafka (any version or even any build you want) clusters (and Zookeeper) with CloudFormation on AWS https://github.com/stealthly/minotaur It is very nice/handy you basically specify your instance types, counts, versions of code, etc and hit a enter https://github.com/stealthly/minotaur/tree/master/labs/kafka e.g. ./minotaur.py lab deploy kafka -e bdoss-dev -d testing -r us-east-1 -z us-east-1a -k http://example.com/kafka.tar.gz -n 3 -i m1.small There is some setup for the bastion host ( https://github.com/stealthly/minotaur/tree/master/infrastructure/aws/bastion ) and supervisor ( https://github.com/stealthly/minotaur/tree/master/supervisor) and after that it is really nice and easy. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Wed, Jan 14, 2015 at 2:54 PM, Joseph Lawson jlaw...@roomkey.com wrote: We have a separate daemon process that assigns EIPs to servers when they startup in an autoscaling group based off of an autoscaling message. So for a cluster of 3 we have 3 EIPs. Then we inject the EIPs into startup script for Kafka which checks to see if it has one of the EIPs and assigns itself the index of that IP so in the list: 10.0.0.1 10.0.0.2 10.0.0.3 1 is broker 0, 2 is broker 1 and 3 is broker 2. All this is injected via cloudformation and then we have a mod value so if we want to spin brokers in the same group we do mod 1,2 and get brokers mod * 3 + index to determine which is in the group. (the EIPs are different as it is a different cloudformation) For redundancy make sure you run at least two that have full replicas of all other partitions. We run replication factor of 3 with three instances so if any goes down the other two bring it back in sync once a fresh server spins in the autoscaling group. From: Dillian Murphey crackshotm...@gmail.com Sent: Wednesday, January 14, 2015 2:42 PM To: users@kafka.apache.org Subject: kafka cluster on aws I can't seem to find much information to help me (being green to kafka) on setting up a cluster on aws. Does anyone have any sources? The question I have off the bat is, what methods have already been explored to generate a unique broker id? If I spin up a new server, do I just need to maintain my own broker-id list somewhere so I don't re-use an already allocated broker id? Also, I read an article about a broker going down and requiring a new broker be spun up with the same id. Is this also something I need to implement? I want to setup a kafka auto-scaling group on AWS, so I can add brokers at well or based on load. It doesn't seem too complicated, or maybe I'm too green to see it, but I don't want to re-invent everything myself. I know Loggly uses AWS/Kafka, so I'm hunting for more details on that too. Thanks for any help
