apparently when ZOO_DATADIR is empty mkdir fubars mkdir ''
when ZOO_DATADIR is empty add this export ZOO_DATADIR="zoo_datadir"
if [ -z "$ZOOPIDFILE" ]; then
ZOO_DATADIR="$(grep "^[[:space:]]*dataDir" "$ZOOCFG" | sed -e 's/.*=//')"
if [ ! -d "$ZOO_DATADIR" ]; then
echo "ZOO_DATADIR="
export ZOO_DATADIR="zoo_datadir"
echo $ZOO_DATADIR
mkdir -p "$ZOO_DATADIR"
fi
HTH
M-
________________________________
From: Zac Harvey <zac.har...@welltok.com>
Sent: Thursday, December 1, 2016 11:03 AM
To: Martin Gainty; user@zookeeper.apache.org
Subject: Re: Messages intermittently get lost
Thanks everyone
The script can be found like so:
wget
http://apache.spinellicreations.com/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
tar xfz zookeeper-3.4.6.tar.gz
cd zookeeper-3.4.6/bin
zkServer.sh
Let me know if you have any other questions/concerns!
Best,
Zac
________________________________
From: Martin Gainty <mgai...@hotmail.com>
Sent: Wednesday, November 30, 2016 7:57:37 PM
To: user@zookeeper.apache.org; Zac Harvey
Subject: Re: Messages intermittently get lost
absolutely..lets get this resolved asap
Zac...can you send me the abending script?
Molte Grazie Flavio!
Martini
______________________________________________
________________________________
From: Flavio Junqueira <f...@apache.org>
Sent: Wednesday, November 30, 2016 6:41 PM
To: user@zookeeper.apache.org
Cc: us...@kafka.apache.org
Subject: Re: Messages intermittently get lost
Hey Martin,
Do you want to create a jira for this and report the issue?
-Flavio
> On 30 Nov 2016, at 18:33, Martin Gainty <mgai...@hotmail.com> wrote:
>
> a shock when a zk script goes fubar
>
>
> zookeeper devs can we get some help for sh zkServer.sh status?
>
>
> Thanks!
>
> Martin
> ______________________________________________
>
>
>
> ________________________________
> From: Zac Harvey <zac.har...@welltok.com>
> Sent: Wednesday, November 30, 2016 10:25 AM
> To: us...@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Martin, makes sense.
>
>
> When I SSH into all 3 of my ZK nodes and run:
>
>
> sh zkServer.sh status
>
>
> All three of them give me the following output:
>
>
> JMX enabled by default
>
> zkServer.sh: 81: /opt/zookeeper/bin/zkEnv.sh: Syntax error: "(" unexpected
> (expecting "fi")
>
>
> Looks like a bug in the ZK shell script?
>
>
> Best,
>
> Zac
>
> ________________________________
> From: Martin Gainty <mgai...@hotmail.com>
> Sent: Tuesday, November 29, 2016 11:18:21 AM
> To: us...@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Zach
>
>
> we dont know whats causing this intermittent problem..so lets Divide and
> Conquer each part of this problem individually starting at the source of the
> data feeds
>
>
> Let us eliminate any potential problem with feeds from external sources
>
>
> Once you verify the zookeeper feeds are 100% reliable lets move onto Kafka
>
>
> Pingback when you have verifiable results from zookeeper feeds
>
>
> Thanks
>
> Martin
> ______________________________________________
>
>
>
> ________________________________
> From: Zac Harvey <zac.har...@welltok.com>
> Sent: Tuesday, November 29, 2016 10:46 AM
> To: us...@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Does anybody have any idea why ZK might be to blame if messages sent by a
> Kafka producer fail to be received by a Kafka consumer?
>
> ________________________________
> From: Zac Harvey <zac.har...@welltok.com>
> Sent: Monday, November 28, 2016 9:07:41 AM
> To: us...@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Thanks Martin, I will look at those links.
>
>
> But you seem to be 100% confident that the problem is with ZooKeeper...can I
> ask why? What is it about my problem description that makes you think this is
> an issue with ZooKeeper?
>
> ________________________________
> From: Martin Gainty <mgai...@hotmail.com>
> Sent: Friday, November 25, 2016 1:46:28 PM
> To: us...@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
>
>
> ________________________________
> From: Zac Harvey <zac.har...@welltok.com>
> Sent: Friday, November 25, 2016 6:17 AM
> To: us...@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Martin,
>
>
> My server.properties looks like this:
>
>
> listeners=PLAINTEXT://0.0.0.0:9092
>
> advertised.host.name=<SERVER_PUBLIC_DNS>
>
> broker.id=2
>
> port=9092
>
> num.partitions=4
>
> zookeeper.connect=zkA:2181,zkB:2181,zkC:2181
>
> MG>can you check status for each ZK Node in the quorum?
>
> sh>$ZOOKEEPER_HOME/bin/zkServer.sh status
>
> http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html
ZooKeeper problems and solutions -
IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with
Apache ZooKeeper.
> ZooKeeper problems and solutions -
> IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
ZooKeeper problems and solutions -
IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with
Apache ZooKeeper.
> www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to
creating innovations that matter for the world
> Use these solutions to resolve the problems that you might encounter with
> Apache ZooKeeper.
>
>
>
> ZooKeeper problems and solutions -
> IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
ZooKeeper problems and solutions -
IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with
Apache ZooKeeper.
> ZooKeeper problems and solutions -
> IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
ZooKeeper problems and solutions -
IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with
Apache ZooKeeper.
> www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to
creating innovations that matter for the world
> Use these solutions to resolve the problems that you might encounter with
> Apache ZooKeeper.
>
>
>
> www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to
creating innovations that matter for the world
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to
creating innovations that matter for the world
> www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to
creating innovations that matter for the world
> For more than a century IBM has been dedicated to every client's success and
> to creating innovations that matter for the world
>
>
>
> Use these solutions to resolve the problems that you might encounter with
> Apache ZooKeeper.
>
>
>
>
> ZooKeeper problems and solutions -
> IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
ZooKeeper problems and solutions -
IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with
Apache ZooKeeper.
> ZooKeeper problems and solutions -
> IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
ZooKeeper problems and solutions -
IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with
Apache ZooKeeper.
> www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to
creating innovations that matter for the world
> Use these solutions to resolve the problems that you might encounter with
> Apache ZooKeeper.
>
>
>
> ZooKeeper problems and solutions -
> IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
ZooKeeper problems and solutions -
IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with
Apache ZooKeeper.
> ZooKeeper problems and solutions -
> IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
ZooKeeper problems and solutions -
IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with
Apache ZooKeeper.
> www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to
creating innovations that matter for the world
> Use these solutions to resolve the problems that you might encounter with
> Apache ZooKeeper.
>
>
>
> www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to
creating innovations that matter for the world
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to
creating innovations that matter for the world
> www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to
creating innovations that matter for the world
> For more than a century IBM has been dedicated to every client's success and
> to creating innovations that matter for the world
>
>
>
> Use these solutions to resolve the problems that you might encounter with
> Apache ZooKeeper.
>
>
>
> www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to
creating innovations that matter for the world
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> www.ibm.com<http://www.ibm.com>
> For more than a century IBM has been dedicated to every client's success and
> to creating innovations that matter for the world
>
>
>
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> www.ibm.com<http://www.ibm.com>
> For more than a century IBM has been dedicated to every client's success and
> to creating innovations that matter for the world
>
>
>
> www.ibm.com<http://www.ibm.com>
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> www.ibm.com<http://www.ibm.com>
> For more than a century IBM has been dedicated to every client's success and
> to creating innovations that matter for the world
>
>
>
> For more than a century IBM has been dedicated to every client's success and
> to creating innovations that matter for the world
>
>
>
> Use these solutions to resolve the problems that you might encounter with
> Apache ZooKeeper.
>
>
>
> MG>*greetings from Plimoth Mass*
>
> MG>M
>
>
> num.network.threads=3
>
> num.io.threads=8
>
> socket.send.buffer.bytes=102400
>
> socket.receive.buffer.bytes=102400
>
> log.dirs=/tmp/kafka-logs
>
> num.recovery.threads.per.data.dir=1
>
> log.retention.hours=168
>
> log.segment.bytes=1073741824
>
> log.retention.check.interval.ms=300000
>
> zookeeper.connection.timeout.ms=6000
>
> delete.topic.enable=true
>
> auto.leader.rebalance.enable=true
>
>
> Above, 'zkA', 'zkB' and 'zkC' are defined in /etc/hosts and are valid ZK
> servers, and <SERVER_PUBLIC_DNS> is the public DNS of the EC2 (AWS) node that
> this Kafka is running on.
>
>
> Anything look incorrect to you?
>
>
> And yes, yesterday was a holiday, but there is only work! I'll celebrate one
> big, long holiday when I'm dead!
>
>
> Thanks for any-and-all input here!
>
>
> Best,
>
> Zac
>
>
> ________________________________
> From: Martin Gainty <mgai...@hotmail.com>
> Sent: Thursday, November 24, 2016 9:03:33 AM
> To: us...@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Zach
>
>
> there is a rumour that today thursday is a holiday?
>
> in server.properties how are you configuring your server?
>
>
> specifically what are these attributes?
>
>
> num.network.threads=
>
>
> num.io.threads=
>
>
> socket.send.buffer.bytes=
>
>
> socket.receive.buffer.bytes=
>
>
> socket.request.max.bytes=
>
>
> num.partitions=
>
>
> num.recovery.threads.per.data.dir=
>
>
> ?
>
> Martin
> ______________________________________________
>
>
>
> ________________________________
> From: Zac Harvey <zac.har...@welltok.com>
> Sent: Thursday, November 24, 2016 7:05 AM
> To: us...@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Anybody?!? This is very disconcerting!
>
> ________________________________
> From: Zac Harvey <zac.har...@welltok.com>
> Sent: Wednesday, November 23, 2016 5:07:45 AM
> To: us...@kafka.apache.org
> Subject: Messages intermittently get lost
>
> I am playing around with Kafka and have a simple setup:
>
>
> * 1-node Kafka (Ubuntu) server
>
> * 3-node ZK cluster (each on their own Ubuntu server)
>
>
> I have a consumer written in Scala and am using the kafka-console-producer
> (v0.10) that ships with the distribution.
>
>
> I'd say about 20% of the messages I send via the producer never get consumed
> by the Scala process (which is running continuously). No errors on either
> side (producer or consumer): the producer sends, and, nothing...
>
>
> Any ideas as to what might be going on here, or how I could start
> troubleshooting?
>
>
> Thanks!
