Hi Martin, makes sense.
When I SSH into all 3 of my ZK nodes and run: sh zkServer.sh status All three of them give me the following output: JMX enabled by default zkServer.sh: 81: /opt/zookeeper/bin/zkEnv.sh: Syntax error: "(" unexpected (expecting "fi") Looks like a bug in the ZK shell script? Best, Zac ________________________________ From: Martin Gainty <mgai...@hotmail.com> Sent: Tuesday, November 29, 2016 11:18:21 AM To: users@kafka.apache.org Subject: Re: Messages intermittently get lost Hi Zach we dont know whats causing this intermittent problem..so lets Divide and Conquer each part of this problem individually starting at the source of the data feeds Let us eliminate any potential problem with feeds from external sources Once you verify the zookeeper feeds are 100% reliable lets move onto Kafka Pingback when you have verifiable results from zookeeper feeds Thanks Martin ______________________________________________ ________________________________ From: Zac Harvey <zac.har...@welltok.com> Sent: Tuesday, November 29, 2016 10:46 AM To: users@kafka.apache.org Subject: Re: Messages intermittently get lost Does anybody have any idea why ZK might be to blame if messages sent by a Kafka producer fail to be received by a Kafka consumer? ________________________________ From: Zac Harvey <zac.har...@welltok.com> Sent: Monday, November 28, 2016 9:07:41 AM To: users@kafka.apache.org Subject: Re: Messages intermittently get lost Thanks Martin, I will look at those links. But you seem to be 100% confident that the problem is with ZooKeeper...can I ask why? What is it about my problem description that makes you think this is an issue with ZooKeeper? ________________________________ From: Martin Gainty <mgai...@hotmail.com> Sent: Friday, November 25, 2016 1:46:28 PM To: users@kafka.apache.org Subject: Re: Messages intermittently get lost ________________________________ From: Zac Harvey <zac.har...@welltok.com> Sent: Friday, November 25, 2016 6:17 AM To: users@kafka.apache.org Subject: Re: Messages intermittently get lost Hi Martin, My server.properties looks like this: listeners=PLAINTEXT://0.0.0.0:9092 advertised.host.name=<SERVER_PUBLIC_DNS> broker.id=2 port=9092 num.partitions=4 zookeeper.connect=zkA:2181,zkB:2181,zkC:2181 MG>can you check status for each ZK Node in the quorum? sh>$ZOOKEEPER_HOME/bin/zkServer.sh status http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html> www.ibm.com<http://www.ibm.com> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper. ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html> www.ibm.com<http://www.ibm.com> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper. www.ibm.com<http://www.ibm.com> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/> IBM - United States<http://www.ibm.com/> www.ibm.com<http://www.ibm.com> For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper. MG>*greetings from Plimoth Mass* MG>M num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 log.dirs=/tmp/kafka-logs num.recovery.threads.per.data.dir=1 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 zookeeper.connection.timeout.ms=6000 delete.topic.enable=true auto.leader.rebalance.enable=true Above, 'zkA', 'zkB' and 'zkC' are defined in /etc/hosts and are valid ZK servers, and <SERVER_PUBLIC_DNS> is the public DNS of the EC2 (AWS) node that this Kafka is running on. Anything look incorrect to you? And yes, yesterday was a holiday, but there is only work! I'll celebrate one big, long holiday when I'm dead! Thanks for any-and-all input here! Best, Zac ________________________________ From: Martin Gainty <mgai...@hotmail.com> Sent: Thursday, November 24, 2016 9:03:33 AM To: users@kafka.apache.org Subject: Re: Messages intermittently get lost Hi Zach there is a rumour that today thursday is a holiday? in server.properties how are you configuring your server? specifically what are these attributes? num.network.threads= num.io.threads= socket.send.buffer.bytes= socket.receive.buffer.bytes= socket.request.max.bytes= num.partitions= num.recovery.threads.per.data.dir= ? Martin ______________________________________________ ________________________________ From: Zac Harvey <zac.har...@welltok.com> Sent: Thursday, November 24, 2016 7:05 AM To: users@kafka.apache.org Subject: Re: Messages intermittently get lost Anybody?!? This is very disconcerting! ________________________________ From: Zac Harvey <zac.har...@welltok.com> Sent: Wednesday, November 23, 2016 5:07:45 AM To: users@kafka.apache.org Subject: Messages intermittently get lost I am playing around with Kafka and have a simple setup: * 1-node Kafka (Ubuntu) server * 3-node ZK cluster (each on their own Ubuntu server) I have a consumer written in Scala and am using the kafka-console-producer (v0.10) that ships with the distribution. I'd say about 20% of the messages I send via the producer never get consumed by the Scala process (which is running continuously). No errors on either side (producer or consumer): the producer sends, and, nothing... Any ideas as to what might be going on here, or how I could start troubleshooting? Thanks!