On Thu, Apr 29, 2010 at 9:49 AM, Patrick Hunt <ph...@apache.org> wrote:
> Is there any good (simple/fast/bulletproof) way to monitor the FD use inside
> the jvm? If so we could stop accepting new client connections once we get
> close to the os imposed limit... The test would have to be a bulletproof one
> though - we wouldn't want to end up in some worse situation (where we refuse
> connection because we mistakenly believe that the limit has been reached).
> Might be good to open a JIRA for this and add some tests. In particular we
> should verify the server handles this as gracefully as it can when the limit
> has been reached.
Poking around with jconsole I found two stats that already measure FDs:
They're described (rather tersely) at:
So it sounds like the feature request would be stop accepting new
client connections if OpenFileDescriptorCount > 95% of
MaxFileDescriptorCount? Only start accepting new requests when
OpenFileDescriptorCount < 90% of MaxFileDescriptorCount. Basically the
high/low watermark thing.
> On 04/29/2010 09:34 AM, Mahadev Konar wrote:
>> Hi Travis,
>> How many clients did you have connected to this server? Usually the
>> is 8K file descriptors. Did you have clients more than that?
>> Also, if clients fail to attach to a server, they will run off to another
>> server. We do not do any blacklisting because we expect the server to heal
>> and if it does not, it mostly shuts itself down in most of the cases.
>> On 4/29/10 12:08 AM, "Travis Crawford"<traviscrawf...@gmail.com> wrote:
>>> Hey zookeeper gurus -
>>> We recently had a zookeeper outage when one ZK server was started with
>>> a low limit after upgrading to 3.3.0. Several days later the outage
>>> occurred when that node reached its file descriptor limit and clients
>>> started having major issues.
>>> Are there any circumstances when a ZK server will get blacklisted from
>>> the ensemble? Something similar to how tasktrackers are blacklisted
>>> when too many tasks fail.