Re: Zookeeper crashes with EOF Exception

[Bob Sheehan]

unsubscribe


On 2/27/19, 7:32 PM, "zoo_js"  wrote:

Thanks for your response.

ZKeeper crashes when trying to create 84151 th key. I will delete the
snapshots and logs, and will run the whole load testing again. 

I am running on version 3.4.13 in AWS Lightsail, ubuntu system with 2GB Ram,
60GB SD and 1CPU. 
Is it good idea to delete the  snapshot/log periodically ? I tried the
./zkCleanup.sh -n 3 but does not seem to do any deletion of log / snapshot,
please help me out.




--
Sent from: 
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fzookeeper-user.578899.n2.nabble.com%2Fdata=02%7C01%7Cbsheehan%40vmware.com%7C2fe313748a044d7e077008d69d2d5f1b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C636869215529974144sdata=GNJTzAYZRBSNBEFXCv3gLaC%2BOY9NG2W3JOGWCG8O0jg%3Dreserved=0

Re: Zookeeper crashes with EOF Exception

[zoo_js]

Thanks for your response.

ZKeeper crashes when trying to create 84151 th key. I will delete the
snapshots and logs, and will run the whole load testing again. 

I am running on version 3.4.13 in AWS Lightsail, ubuntu system with 2GB Ram,
60GB SD and 1CPU. 
Is it good idea to delete the  snapshot/log periodically ? I tried the
./zkCleanup.sh -n 3 but does not seem to do any deletion of log / snapshot,
please help me out.




--
Sent from: http://zookeeper-user.578899.n2.nabble.com/

Re: Zookeeper crashes with EOF Exception

[Norbert Kalmar]

Sounds like your snapshot is corrupted. But you said ZK is running fine for
some amount of time then crashes?
Maybe it's an invalid PROPOSE message.
By the way, sounds a bit similar to this issue:
https://issues.apache.org/jira/browse/ZOOKEEPER-1955

If it is possible, delete the snapshot and txn logs from data dir (you will
lose your data!) and restart the clusters.
Which version of ZK are you using?

On Wed, Feb 27, 2019 at 12:29 PM zoo_js 
wrote:

> There is 3 snapshot files with 1.01 GB size, each file at around 330 MB of
> size. I have a 56GB of hard disk space available.
>
>
>
> --
> Sent from: http://zookeeper-user.578899.n2.nabble.com/
>

Re: Memory progressive spike using zookeeper HA Mode

[Enrico Olivelli]

JS,
Which version of zookeeper server and client are you using?
Are you using default configuration?

Enrico

Il mer 27 feb 2019, 10:01 zoo_js  ha
scritto:

> Hi,
>
> I have 3 zookeeper servers for HA and was trying from load testing with
> Hashicorp Vault. The process will create 1000 transit keys for encryption
> every 1 minute and goes on, initially zookeeper memory usage will be low
> and
> as it progresses around 50,000 keys memory usage of zookeeper (java) goes
> more than 90% and then crashes, which stops the whole vault engine.
> Attached
> the memory usage snapshot. Please help..
>
> <
> http://zookeeper-user.578899.n2.nabble.com/file/t396600/zookeeper_mem_usage.png>
>
>
> thanks
> JS
>
>
>
> --
> Sent from: http://zookeeper-user.578899.n2.nabble.com/
>

Re: Zookeeper crashes with EOF Exception

[zoo_js]

There is 3 snapshot files with 1.01 GB size, each file at around 330 MB of
size. I have a 56GB of hard disk space available. 



--
Sent from: http://zookeeper-user.578899.n2.nabble.com/

Re: Zookeeper crashes with EOF Exception

[zoo_js]

I am not sure about reading the snapshot, can you help me with the command /
steps to do the same ? I did not run out of disk space, the machine has 60gb
of space. 

thanks 
JS



--
Sent from: http://zookeeper-user.578899.n2.nabble.com/

Re: Zookeeper crashes with EOF Exception

[Norbert Kalmar]

Hi JS,

Looks like there was a Leader election, and during sync phase
(syncWithLeader), the follower tried to deserialize the snapshot, but it is
an incomplete file, hence the EOF exception.
How big is your snapshot? Did you run out of disc space?
Also worth checking for fsync warnings / errors in the log.

Hope this helps.

Regards,
Norbert

On Wed, Feb 27, 2019 at 8:05 AM zoo_js 
wrote:

> Hi all,
>
> We have a 3 node zookeeper cluster used for Vault as HA.  Starting a few
> days ago, the entire cluster crashes a few times per day, all nodes at the
> exact same time. We are running some load test using vault for Data
> encryption. Per minute 1000 keys unique keys will be generated, Once the
> issue started around 270,000 keys.
>
> The following exception is got from the syslog, not sure what's causing
> this
> crash. Please help to proceed..
>
> 2019-02-26 22:35:18,831 [myid:1] - WARN
> [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:Follower@90] - Exception when
> following the leader
> java.io.EOFException
>at
> java.base/java.io.DataInputStream.readFully(DataInputStream.java:202)
>at
> java.base/java.io.DataInputStream.readFully(DataInputStream.java:170)
>at
> org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:94)
>at
> org.apache.zookeeper.server.DataNode.deserialize(DataNode.java:165)
>at
> org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
>at
> org.apache.zookeeper.server.DataTree.deserialize(DataTree.java:1076)
>at
>
> org.apache.zookeeper.server.util.SerializeUtils.deserializeSnapshot(SerializeUtils.java:130)
>at
>
> org.apache.zookeeper.server.ZKDatabase.deserializeSnapshot(ZKDatabase.java:452)
>at
> org.apache.zookeeper.server.quorum.Learner.syncWithLeader(Learner.java:340)
>at
> org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:83)
>at
> org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:981)
> 2019-02-26 22:35:19,349 [myid:1] - INFO
> [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:Follower@169] - shutdown called
> java.lang.Exception: shutdown Follower
>at
> org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:169)
>
> thanks
> JS
>
>
>
>
>
>
> --
> Sent from: http://zookeeper-user.578899.n2.nabble.com/
>

Memory progressive spike using zookeeper HA Mode

[zoo_js]

Hi,

I have 3 zookeeper servers for HA and was trying from load testing with
Hashicorp Vault. The process will create 1000 transit keys for encryption
every 1 minute and goes on, initially zookeeper memory usage will be low and
as it progresses around 50,000 keys memory usage of zookeeper (java) goes
more than 90% and then crashes, which stops the whole vault engine. Attached
the memory usage snapshot. Please help..


 

thanks
JS



--
Sent from: http://zookeeper-user.578899.n2.nabble.com/

Re: Configuring pseudo distributed zookeeper, followers and leader

[Norbert Kalmar]

Hi Weiqi,

How do you start the instances? My theory is that the 2 nodes on the VM
establishes the connection faster than you local machine with the two VM.
So since they already have quorum they will elect the Leader amongst them.
you could start 1 node from VM and on your localhost, and then just wait
until your localhost is the Leader. Then start the 3rd node on the VM.
There really isn't a way to define Leader at startup as much as I can tell.

Hope this helps.

Regards,
Norbert

On Tue, Feb 26, 2019 at 4:46 PM 徐炜淇  wrote:

> Hi,
> I configured a pseudo distributed zookeeper, a master two virtual
> machines. The content of file A are as follows:
> tickTime=2000
> initLimit=10
> syncLimit=5
> dataDir=/home/v7/RyaInstall/zookeeper-3.4.12/data
> clientPort=2181
> server.1=192.168.122.1:2888:3888
> server.2=192.168.122.92:2888:3888
> server.3=192.168.122.152:2888:3888
>
>
>
> The server is my local computer, server 2 and 3 are ,y virtual machines.
> But when I start zookeerper, my localhost always becomes follower, one of
> server2 or server3 becomes the leader.
> I do not know how to make my local computer becomes the leader.
> Can you help me?
>
>
> Best
> Weiqi
>
>
>
>
>
>
>