[
https://issues.apache.org/jira/browse/ZOOKEEPER-885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13999334#comment-13999334
]
Michi Mutsuzaki commented on ZOOKEEPER-885:
-------------------------------------------
I recently tested this with 3.4.5. On my setup with 6 second session timeout,
the client doesn't get disconnected even with 100% disk utilization (using
iostat) when there is no write traffic to ZooKeeper. When there is write
traffic, the connection starts becoming flaky around 80~90% disk utilization.
We could make the disconnect timeout (2/3 of the session timeout) configurable.
The client can keep sending ping to avoid session expiry even if the ping
responses get blocked behind a slow write because the session timestamp gets
updated as soon as the server receives the ping request. You could also have an
additional mechanism for detecting server failures using a 4 letter word
command, although this would put additional load to the server.
I don't have bandwidth to do this for 3.5.0 though. Let me know if anybody is
interested in working on this. Otherwise, I'll push this to 3.6.0.
> Zookeeper drops connections under moderate IO load
> --------------------------------------------------
>
> Key: ZOOKEEPER-885
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-885
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
> Affects Versions: 3.2.2, 3.3.1
> Environment: Debian (Lenny)
> 1Gb RAM
> swap disabled
> 100Mb heap for zookeeper
> Reporter: Alexandre Hardy
> Priority: Critical
> Fix For: 3.5.0
>
> Attachments: WatcherTest.java, benchmark.csv, tracezklogs.tar.gz,
> tracezklogs.tar.gz, zklogs.tar.gz
>
>
> A zookeeper server under minimum load, with a number of clients watching
> exactly one node will fail to maintain the connection when the machine is
> subjected to moderate IO load.
> In a specific test example we had three zookeeper servers running on
> dedicated machines with 45 clients connected, watching exactly one node. The
> clients would disconnect after moderate load was added to each of the
> zookeeper servers with the command:
> {noformat}
> dd if=/dev/urandom of=/dev/mapper/nimbula-test
> {noformat}
> The {{dd}} command transferred data at a rate of about 4Mb/s.
> The same thing happens with
> {noformat}
> dd if=/dev/zero of=/dev/mapper/nimbula-test
> {noformat}
> It seems strange that such a moderate load should cause instability in the
> connection.
> Very few other processes were running, the machines were setup to test the
> connection instability we have experienced. Clients performed no other read
> or mutation operations.
> Although the documents state that minimal competing IO load should present on
> the zookeeper server, it seems reasonable that moderate IO should not cause
> problems in this case.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
