Re: Permission denied as zookeeper user

2018-07-17 Thread adrien ruffie
Thank Andor !
I have found thé problème.
My zookeeper user doesn't have right on /Var/lib/zookeeper and 
/usr/local/zookeeper
Because directoires were créateur as Root !

Télécharger Outlook pour Android



De : Andor Molnar
Envoyé : mardi 17 juillet 21:04
Objet : Re: Permission denied as zookeeper user
À : user@zookeeper.apache.org


Hi Adrian, Check the ‘dataDir’ and ‘dataLogDir’ (might be the same) directory 
permissions. Zookeeper running user must have write permissions to them. 
Regards, Andor > On 2018. Jul 17., at 18:39, adrien ruffie wrote: > > Hello 
everyone, > > > I have a little problem when I try to launch "./zkServer.sh 
start" with my "zookeeper" user ... > > I get the following stacktrace: > > > 
2018-07-17 18:26:53,569 [myid:5] - INFO 
[WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message 
format version), 3 (n.leader), 0x276590006 (n.zxid), 0x43 (n.round), 
FOLLOWING (n.state), 1 (n.sid), 0x2765a (n.peerEpoch) FOLLOWING (my state) > 
2018-07-17 18:26:53,574 [myid:5] - INFO 
[QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:2181:Learner@326] - Getting a diff from the 
leader 0x276590006 > 2018-07-17 18:26:53,575 [myid:5] - WARN 
[QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:2181:Follower@87] - Exception when 
following the leader > java.io.IOException: Permission denied > at 
java.io.UnixFileSystem.createFileExclusively(Native Method) > at 
java.io.File.createNewFile(File.java:1012) > at 
org.apache.zookeeper.server.quorum.Learner.syncWithLeader(Learner.java:436) > 
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:80) > 
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:846) > 
2018-07-17 18:26:53,575 [myid:5] - INFO 
[QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:2181:Follower@164] - shutdown called > 
java.lang.Exception: shutdown Follower > at 
org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:164) > at 
org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:850) > > > > 
> > But when I launch as root, the problem disappears ... > > I understand 
which is a permission denied for my zookeeper user, but the stack doesn't give 
me, the concerned file or directory ... > > > Do you already get the same 
problem ? > > > Thank, and best regards, > > > Adrien



Permission denied as zookeeper user

2018-07-17 Thread adrien ruffie
Hello everyone,


I have a little problem when I try to launch "./zkServer.sh start" with my 
"zookeeper" user ...

I get the following stacktrace:


2018-07-17 18:26:53,569 [myid:5] - INFO  
[WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message 
format version), 3 (n.leader), 0x276590006 (n.zxid), 0x43 (n.round), 
FOLLOWING (n.state), 1 (n.sid), 0x2765a (n.peerEpoch) FOLLOWING (my state)
2018-07-17 18:26:53,574 [myid:5] - INFO  
[QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:2181:Learner@326] - Getting a diff from the 
leader 0x276590006
2018-07-17 18:26:53,575 [myid:5] - WARN  
[QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:2181:Follower@87] - Exception when 
following the leader
java.io.IOException: Permission denied
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(File.java:1012)
at org.apache.zookeeper.server.quorum.Learner.syncWithLeader(Learner.java:436)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:80)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:846)
2018-07-17 18:26:53,575 [myid:5] - INFO  
[QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:2181:Follower@164] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:164)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:850)





But when I launch as root, the problem disappears ...

I understand which is a permission denied for my zookeeper user, but the stack 
doesn't give me, the concerned file or directory ...


Do you already get the same problem ?


Thank, and best regards,


Adrien


Re: Zookeeper consistency

2018-07-17 Thread Michael Han
>> I think that Zookeeper is linearable only if there are only write
operation.

Yes writes are linearizable because writes are totally ordered globally.

For read, a linearizable read needs to read the latest writes in the system
at the point in time the read is issued; so by this definition, ZK read is
not linearizable, even with sync+read as both are separate operations. If
we provide a quorum read (so read operations are broadcasted globally as
well), then the read will be linearizable as well, but no practical systems
would do that due to performance reasons.


On Tue, Jul 17, 2018 at 3:13 AM, chen dongming  wrote:

> hi,
>
> I am reading the old thread of
> http://comments.gmane.org/gmane.comp.java.hadoop.zookeeper.user/5221.
>
> I have 2 questions.
>
>
> Question 1:
>
> I agree with Zookeeper is sequential consistency, but I don't think
> Zookeeper is linearizability even if doing sync+read for every read.
> for example,
>
> timeline >
>
> C1:sync()  r1(x)
>
> C2: w2(x=1) w2(x=2)
>
> It is possible r1 return x=1.  This history is not linearable according
> to Herlihy difination.
>
>
> Question 2:
>
> I think that Zookeeper is linearable only if there are only write
> operation.
>
>
> Dominic,
>
> Best Regard.
>
>
>


Re: Permission denied as zookeeper user

2018-07-17 Thread Andor Molnar
Hi Adrian,

Check the ‘dataDir’ and ‘dataLogDir’ (might be the same) directory permissions.
Zookeeper running user must have write permissions to them.

Regards,
Andor



> On 2018. Jul 17., at 18:39, adrien ruffie  wrote:
> 
> Hello everyone,
> 
> 
> I have a little problem when I try to launch "./zkServer.sh start" with my 
> "zookeeper" user ...
> 
> I get the following stacktrace:
> 
> 
> 2018-07-17 18:26:53,569 [myid:5] - INFO  
> [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message 
> format version), 3 (n.leader), 0x276590006 (n.zxid), 0x43 (n.round), 
> FOLLOWING (n.state), 1 (n.sid), 0x2765a (n.peerEpoch) FOLLOWING (my state)
> 2018-07-17 18:26:53,574 [myid:5] - INFO  
> [QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:2181:Learner@326] - Getting a diff from 
> the leader 0x276590006
> 2018-07-17 18:26:53,575 [myid:5] - WARN  
> [QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:2181:Follower@87] - Exception when 
> following the leader
> java.io.IOException: Permission denied
> at java.io.UnixFileSystem.createFileExclusively(Native Method)
> at java.io.File.createNewFile(File.java:1012)
> at org.apache.zookeeper.server.quorum.Learner.syncWithLeader(Learner.java:436)
> at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:80)
> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:846)
> 2018-07-17 18:26:53,575 [myid:5] - INFO  
> [QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:2181:Follower@164] - shutdown called
> java.lang.Exception: shutdown Follower
> at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:164)
> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:850)
> 
> 
> 
> 
> 
> But when I launch as root, the problem disappears ...
> 
> I understand which is a permission denied for my zookeeper user, but the 
> stack doesn't give me, the concerned file or directory ...
> 
> 
> Do you already get the same problem ?
> 
> 
> Thank, and best regards,
> 
> 
> Adrien



Re: ZooKeeper Cluster Health Checking

2018-07-17 Thread harish lohar
We did it via java monitoring app , using zookeeper java api which sends 4
lw commands to zookeeper and returns the output.


Thanks
Harish

On Tue, Jul 17, 2018 at 2:00 AM adrien ruffie 
wrote:

> Hi Harish,
>
>
> thank you very much for this advise and explanation !
>
> Do you think with just a simple script shell for checking all this metrics
> is enough ? Or would better to do it in a Java with a simple monitoring
> application?
>
>
> Thank again,
>
>
> Best regards,
>
>
> Adrien
>
> 
> De : harish lohar 
> Envoyé : mardi 17 juillet 2018 04:13:51
> À : user@zookeeper.apache.org
> Objet : Re: ZooKeeper Cluster Health Checking
>
> Hi Adrian,
> Below zookeeper commands are generally used to get health of zookeeper
> cluster
> stat
>
> Lists brief details for the server and connected clients.
>
> usage echo stat | nc server port
>
> This gives whether cluster is up /down. If down this will give that
>
> Zookeeper instance is currently not serving any request -  which means
> either the leader election is failing or <= 50% of zookeeper node in
> cluster are down.
>
>
> mntr
>
> *New in 3.4.0:* Outputs a list of variables that could be used for
> monitoring the health of the cluster.
>
> $ echo mntr | nc localhost 2185
>
> zk_version  3.4.0
> zk_avg_latency  0
> zk_max_latency  0
> zk_min_latency  0
> zk_packets_received 70
> zk_packets_sent 69
> zk_outstanding_requests 0
> zk_server_state leader
> zk_znode_count   4
> zk_watch_count  0
> zk_ephemerals_count 0
> zk_approximate_data_size27
> zk_followers4   - only exposed by the Leader
> zk_synced_followers 4   - only exposed by the Leader
> zk_pending_syncs0   - only exposed by the Leader
> zk_open_file_descriptor_count 23- only available on Unix platforms
> zk_max_file_descriptor_count 1024   - only available on Unix platforms
>
> The output is compatible with java properties format and the content may
> change over time (new keys added). Your scripts should expect changes.
>
> ATTENTION: Some of the keys are platform specific and some of the keys are
> only exported by the Leader.
>
> The output contains multiple lines with the following format:
>
>
> On Mon, Jul 16, 2018 at 10:13 AM adrien ruffie 
> wrote:
>
> > Hello all,
> >
> >
> > In my company we have a Zookeeper production cluster.
> >
> >
> > But we don't really know how can we check the health of our cluster...
> >
> >
> > Can we advise us about this topic ?
> >
> >
> > I know this topic may has been cropping up for a while, but I don't
> really
> > found any concrete solution.
> >
> >
> > Do you use a monitoring tools ? Which can launch alert ?
> >
> > What metrics/properties/any thing which can indicate that our cluster
> > isn't in good health.
> >
> >
> > Thank you very much and best regards
> >
> >
> > Adrien
> >
>


RE: zookeeper as systemd

2018-07-17 Thread adrien ruffie
thank you very much Norbert !

I take your useful suggestion, and try to launch zookeeper with your

service file.


Great thank,


Adrien


De : Norbert Kalmar 
Envoyé : lundi 16 juillet 2018 16:18:36
À : user@zookeeper.apache.org
Objet : Re: zookeeper as systemd

Some prefer to use start-foreground to basically "bypass" the bash script
(and the background management), and make systemd care about it. Also to
ignore any nohup. So while it is also my understanding that the recommended
way is to start ZK with simple start, some use start-foreground to start as
a systemd service.

Type=simple - drawback is depending services (if any) will not know if it
is truly ready (sockets as well)
Type=forking - drawback is there might be some issues with the PID (file).
So you should also define where the pid file
is: PIDFILE="/var/run/zookeeper/zookeeper_server.pid"

Any combinations should work. It really comes down to preferans I think.

Disclaimer: I'm kind of trying to make educated guesses here, so a
Linux-slash-zookeeper guru could probably be more of a help here :)

This is how I would create systemd service file (this time using start not
start-foreground) - or something like that:
Put the systemd service file in: /etc/systemd/system/zookeeper.service

[Unit]

Description=Zookeeper

After=network.target syslog.target


[Service]

SyslogIdentifier=zookeeper

TimeoutStartSec=10min

Type=forking

User=zookeeper

Group=zookeeper

PIDFILE="/var/run/zookeeper/zookeeper_server.pid"

ExecStart=/usr/lib/zookeeper/bin/zkServer.sh start

ExecStop=/usr/lib/zookeeper/bin/zkServer.sh stop

WorkingDirectory=/var/lib/zookeeper


[Install]

WantedBy=multi-user.target


Of course not everything is required (User, Group, WorkingDirectory
possibly optional)


But I would dig deeper before saying anything for sure.


Regards,

Norbert

On Mon, Jul 16, 2018 at 3:01 PM adrien ruffie 
wrote:

> Thank Norbert for this good explanation.
>
> Yes I'm also a really lost here a bit ...
>
> But it's a zookeeper production's cluster of 5 nodes.
> It should be start instead start-foreground isn't it ? (with simple type)
> 
> De : Norbert Kalmar 
> Envoyé : lundi 16 juillet 2018 13:49:16
> À : user@zookeeper.apache.org
> Objet : Re: zookeeper as systemd
>
> The type is Linux config, and forking is used when we want the process to
> call fork() during startup. This should guarantee that when startup is
> finished, all ports are open. In my understanding, using FORKING will tell
> systemd the startup is complete and ports are open. See
> https://www.freedesktop.org/software/systemd/man/systemd.service.html for
> more information.
> TYPE=simple is mainly for daemons that don't use network channels, or they
> have their own socket activation.
>
> So in my opinion, TYPE=forking is better for ZK. But! Forking should
> require a PIDfile in order for it to work properly... but since systemd is
> used, it should know the PID as it is a supervisor. I'm also lost here a
> bit...
> Looking at ZK PID file should be
> available: ZOOPIDFILE="$ZOO_DATADIR/zookeeper_server.pid"
>
>
> About the simple start or start-foreground... I think I was quick to say
> only use simple start in prod environment. This is much more complicated,
> for example start-foreground would just make it ignore nohup.
>
> Doing some search, these 2 questions might help:
>
> https://stackoverflow.com/questions/40620544/systemd-zookeeper-service-failed
>
> https://askubuntu.com/questions/979498/how-to-start-a-zookeeper-daemon-after-booting-under-specific-user-in-ubuntu-serv
>
> Putting this all together, I think if you don't really care when socket is
> ready, just use TYPE=simple and call with start-foreground.
>
> Regards,
> Norbert
>
>
>
> On Mon, Jul 16, 2018 at 11:48 AM adrien ruffie 
> wrote:
>
> > Thank Nobert !
> >
> >
> > It really help me,
> >
> >
> > according to you, what would you recommend?
> >
> >
> > To launch with "Type=simple" and zkServer.sh start ?
> >
> > Or "Type=forking" and zkServer.sh start ?
> >
> >
> > Because start command launch Zookeeper as a Daemon,
> >
> > but if I use "Type=simple" the system already daemonize the process ...
> >
> > Do you think that can be daemonize a daemon ... ? Strange
> >
> >
> > I really ready to use "Type=forking" option but, according to this
> > following post
> >
> >
> > https://bbs.archlinux.org/viewtopic.php?id=191669
> >
> >
> > the "Type=forking" is not really recommended ...
> >
> > what do you think ?
> >
> >
> > Adrien
> >
> >
> > 
> > De : Norbert Kalmar 
> > Envoyé : lundi 16 juillet 2018 10:15
> > À : user@zookeeper.apache.org
> > Objet : Re: zookeeper as systemd
> >
> > Hi Adrien,
> >
> > zkServer.sh start-foreground - starts the ZooKeeper process in the
> > foreground. Good for debugging (thats what I use it for), or check
> > something, as you will have the logs printed to standard output (console
> > most probably).
> > 

RE: ZooKeeper Cluster Health Checking

2018-07-17 Thread adrien ruffie
Hi Harish,


thank you very much for this advise and explanation !

Do you think with just a simple script shell for checking all this metrics is 
enough ? Or would better to do it in a Java with a simple monitoring 
application?


Thank again,


Best regards,


Adrien


De : harish lohar 
Envoyé : mardi 17 juillet 2018 04:13:51
À : user@zookeeper.apache.org
Objet : Re: ZooKeeper Cluster Health Checking

Hi Adrian,
Below zookeeper commands are generally used to get health of zookeeper
cluster
stat

Lists brief details for the server and connected clients.

usage echo stat | nc server port

This gives whether cluster is up /down. If down this will give that

Zookeeper instance is currently not serving any request -  which means
either the leader election is failing or <= 50% of zookeeper node in
cluster are down.


mntr

*New in 3.4.0:* Outputs a list of variables that could be used for
monitoring the health of the cluster.

$ echo mntr | nc localhost 2185

zk_version  3.4.0
zk_avg_latency  0
zk_max_latency  0
zk_min_latency  0
zk_packets_received 70
zk_packets_sent 69
zk_outstanding_requests 0
zk_server_state leader
zk_znode_count   4
zk_watch_count  0
zk_ephemerals_count 0
zk_approximate_data_size27
zk_followers4   - only exposed by the Leader
zk_synced_followers 4   - only exposed by the Leader
zk_pending_syncs0   - only exposed by the Leader
zk_open_file_descriptor_count 23- only available on Unix platforms
zk_max_file_descriptor_count 1024   - only available on Unix platforms

The output is compatible with java properties format and the content may
change over time (new keys added). Your scripts should expect changes.

ATTENTION: Some of the keys are platform specific and some of the keys are
only exported by the Leader.

The output contains multiple lines with the following format:


On Mon, Jul 16, 2018 at 10:13 AM adrien ruffie 
wrote:

> Hello all,
>
>
> In my company we have a Zookeeper production cluster.
>
>
> But we don't really know how can we check the health of our cluster...
>
>
> Can we advise us about this topic ?
>
>
> I know this topic may has been cropping up for a while, but I don't really
> found any concrete solution.
>
>
> Do you use a monitoring tools ? Which can launch alert ?
>
> What metrics/properties/any thing which can indicate that our cluster
> isn't in good health.
>
>
> Thank you very much and best regards
>
>
> Adrien
>


Re: Problem with ZK log files and snapshots

2018-07-17 Thread Baskar Duraikannu
If your transactions are short lived (something like you are taking a lock and 
then releasing it), you will find that “containers” holding these locks are not 
cleaned by Zookeeper. This in turn will make snapshot files big and will cause 
all sorts of issues. I found myself in that situation few months back. New 
version of zookeeper has container support which solves this issue.  We had 
this issue and upgrading to 3.5.4-Beta and appropriate version of curator 
solved the issue. 

Good luck.

> On Jul 16, 2018, at 7:04 AM, Jostein Elvaker Haande  
> wrote:
> 
> Hello,
> 
> I'm using ZooKeeper 3.4.12, and I'm facing an issue where the ZK log files
> keep up eating disk space. I'm running this on several machines, and the
> symptoms I'm seeing are the seem:
> 
> - I see a lot of log files
> - Rarely, if ever any snapshot files
> 
> I started digging into it, and from
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fzookeeper.apache.org%2Fdoc%2Fr3.4.12%2FzookeeperAdmin.html%23sc_maintenance=02%7C01%7C%7Ca930ebc3267246f355f208d5eb0bddf9%7C84df9e7fe9f640afb435%7C1%7C0%7C636673358550132623=pBKODIXN1Yl5%2BL7zAtWvgoocWK20g8BAbFygPFpa648%3D=0
> one should use the ZK Java class org.apache.zookeeper.server.PurgeTxnLog
> for this. However, seeing as I rarely see any snapshot files, PurgeTxnLog
> never performs any cleanup.
> 
> Going deeper down, I can see from the documentation that a snapshot is
> created every 100K transactions that are kept in the log files, so I was
> interested in seeing how man transaction my logs kept. I then made a script
> that used the Java class org.apache.zookeeper.server.LogFormatter which
> outputs number of transactions on the same line, and I noticed that my ZK
> instances saw a very low transaction count. For most of my instances, the
> transaction count was below 20K.
> 
> Now I know that by setting snapCount to a value lower than the default
> 100K, it should generate snapshots more frequently, and ZK should then be
> able to do this cleanup automatically which is controlled
> by autopurge.snapRetainCount and autopurge.purgeInterval.
> 
> However, when setting a new value for snapCount, this does not seem to have
> any effect on the existing transaction log files. So my question is, how do
> I deal with the old transaction logs? How do I proceed to clean this away
> safely?
> 
> -- 
> Yours sincerely Jostein Elvaker Haande
> "A free society is a society where it is safe to be unpopular"
> - Adlai Stevenson
> 
> https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Ftolecnal.net=02%7C01%7C%7Ca930ebc3267246f355f208d5eb0bddf9%7C84df9e7fe9f640afb435%7C1%7C0%7C636673358550132623=kUcc%2BHDxvoBG%2F7D3HQx2XxsPHfDoQmkRtXH6fVlN5To%3D=0
>  -- tolecnal at tolecnal dot net



Zookeeper consistency

2018-07-17 Thread chen dongming
hi,

I am reading the old thread of 
http://comments.gmane.org/gmane.comp.java.hadoop.zookeeper.user/5221.

I have 2 questions.


Question 1:

I agree with Zookeeper is sequential consistency, but I don't think 
Zookeeper is linearizability even if doing sync+read for every read.  
for example,

timeline >

C1:    sync()  r1(x)

C2: w2(x=1) w2(x=2)

It is possible r1 return x=1.  This history is not linearable according 
to Herlihy difination.


Question 2:

I think that Zookeeper is linearable only if there are only write operation.


Dominic,

Best Regard.




Re: ZooKeeper Cluster Health Checking

2018-07-17 Thread Norbert Kalmar
Hi Adrien,

Take a look at monitoring in src/contrib/monitoring - it does what you
would like to achieve, in python. Read the README for more information:
https://github.com/apache/zookeeper/tree/master/src/contrib/monitoring

If this one is not good for you, you can use JMX to query MBeans.

A heads-up: At some point, 4letter words will be deprecated and possibly
removed due to security issues.

Regards,
Norbert

On Tue, Jul 17, 2018 at 8:00 AM adrien ruffie 
wrote:

> Hi Harish,
>
>
> thank you very much for this advise and explanation !
>
> Do you think with just a simple script shell for checking all this metrics
> is enough ? Or would better to do it in a Java with a simple monitoring
> application?
>
>
> Thank again,
>
>
> Best regards,
>
>
> Adrien
>
> 
> De : harish lohar 
> Envoyé : mardi 17 juillet 2018 04:13:51
> À : user@zookeeper.apache.org
> Objet : Re: ZooKeeper Cluster Health Checking
>
> Hi Adrian,
> Below zookeeper commands are generally used to get health of zookeeper
> cluster
> stat
>
> Lists brief details for the server and connected clients.
>
> usage echo stat | nc server port
>
> This gives whether cluster is up /down. If down this will give that
>
> Zookeeper instance is currently not serving any request -  which means
> either the leader election is failing or <= 50% of zookeeper node in
> cluster are down.
>
>
> mntr
>
> *New in 3.4.0:* Outputs a list of variables that could be used for
> monitoring the health of the cluster.
>
> $ echo mntr | nc localhost 2185
>
> zk_version  3.4.0
> zk_avg_latency  0
> zk_max_latency  0
> zk_min_latency  0
> zk_packets_received 70
> zk_packets_sent 69
> zk_outstanding_requests 0
> zk_server_state leader
> zk_znode_count   4
> zk_watch_count  0
> zk_ephemerals_count 0
> zk_approximate_data_size27
> zk_followers4   - only exposed by the Leader
> zk_synced_followers 4   - only exposed by the Leader
> zk_pending_syncs0   - only exposed by the Leader
> zk_open_file_descriptor_count 23- only available on Unix platforms
> zk_max_file_descriptor_count 1024   - only available on Unix platforms
>
> The output is compatible with java properties format and the content may
> change over time (new keys added). Your scripts should expect changes.
>
> ATTENTION: Some of the keys are platform specific and some of the keys are
> only exported by the Leader.
>
> The output contains multiple lines with the following format:
>
>
> On Mon, Jul 16, 2018 at 10:13 AM adrien ruffie 
> wrote:
>
> > Hello all,
> >
> >
> > In my company we have a Zookeeper production cluster.
> >
> >
> > But we don't really know how can we check the health of our cluster...
> >
> >
> > Can we advise us about this topic ?
> >
> >
> > I know this topic may has been cropping up for a while, but I don't
> really
> > found any concrete solution.
> >
> >
> > Do you use a monitoring tools ? Which can launch alert ?
> >
> > What metrics/properties/any thing which can indicate that our cluster
> > isn't in good health.
> >
> >
> > Thank you very much and best regards
> >
> >
> > Adrien
> >
>