Dave Wright commented on ZOOKEEPER-885:

I don't think the cause of this is much of a mystery, as we experienced similar 
problems when we had the zookeeper files on the same filesystem as an IO-heavy 
application that was doing buffered IO. Quite simply, when zookeeper does a 
sync on its own files, it causes the entire filesystem to sync, flushing any 
buffered data from the IO-heavy application and freezing the ZK server process 
for long enough for heartbeats to timeout. 

When you say "moderate IO load" I'm curious what the bottleneck is - the dd 
command will copy data as fast as possible, if you're only getting 4MB/sec, the 
underlying device must be pretty slow, which would further indicate why a 
sync() request would take a while to complete. 

The only fix we've seen is to put the ZK files on their own device, although 
you may be able to fix it with a different partition on the same device. 

> Zookeeper drops connections under moderate IO load
> --------------------------------------------------
>                 Key: ZOOKEEPER-885
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-885
>             Project: Zookeeper
>          Issue Type: Bug
>          Components: server
>    Affects Versions: 3.2.2, 3.3.1
>         Environment: Debian (Lenny)
> 1Gb RAM
> swap disabled
> 100Mb heap for zookeeper
>            Reporter: Alexandre Hardy
>            Priority: Critical
>         Attachments: benchmark.csv, tracezklogs.tar.gz, tracezklogs.tar.gz, 
> WatcherTest.java, zklogs.tar.gz
> A zookeeper server under minimum load, with a number of clients watching 
> exactly one node will fail to maintain the connection when the machine is 
> subjected to moderate IO load.
> In a specific test example we had three zookeeper servers running on 
> dedicated machines with 45 clients connected, watching exactly one node. The 
> clients would disconnect after moderate load was added to each of the 
> zookeeper servers with the command:
> {noformat}
> dd if=/dev/urandom of=/dev/mapper/nimbula-test
> {noformat}
> The {{dd}} command transferred data at a rate of about 4Mb/s.
> The same thing happens with
> {noformat}
> dd if=/dev/zero of=/dev/mapper/nimbula-test
> {noformat}
> It seems strange that such a moderate load should cause instability in the 
> connection.
> Very few other processes were running, the machines were setup to test the 
> connection instability we have experienced. Clients performed no other read 
> or mutation operations.
> Although the documents state that minimal competing IO load should present on 
> the zookeeper server, it seems reasonable that moderate IO should not cause 
> problems in this case.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

Reply via email to