[
https://issues.apache.org/jira/browse/KAFKA-2580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14963986#comment-14963986
]
Todd Palino commented on KAFKA-2580:
------------------------------------
I agree with [~jkreps] here, that having a high FD limit is not a bad thing. As
[~jjkoshy] noted, we're already running at 400k internally (recently increased
from 200k). Part of that is to handle growth, and part of that is to have a
good bit of headroom if something starts to leak FDs so we have some time to
address it before it kills the process (we alert at 50% utilization).
The LRU cache option is probably the best. You can set it to an arbitrarily
high number (the best option here might be to cap it near, but below, your
per-process limit) if you want to effectively disable it, and it would
generally avoid the process of having to check and act on expiring the FDs in
the timer option. I can see arguments for setting the default either high or
low (and I consider 10k to be low). Regardless, as long as it's configurable
and documented it will be fine.
> Kafka Broker keeps file handles open for all log files (even if its not
> written to/read from)
> ---------------------------------------------------------------------------------------------
>
> Key: KAFKA-2580
> URL: https://issues.apache.org/jira/browse/KAFKA-2580
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8.2.1
> Reporter: Vinoth Chandar
> Assignee: Grant Henke
>
> We noticed this in one of our clusters where we stage logs for a longer
> amount of time. It appears that the Kafka broker keeps file handles open even
> for non active (not written to or read from) files. (in fact, there are some
> threads going back to 2013
> http://grokbase.com/t/kafka/users/132p65qwcn/keeping-logs-forever)
> Needless to say, this is a problem and forces us to either artificially bump
> up ulimit (its already at 100K) or expand the cluster (even if we have
> sufficient IO and everything).
> Filing this ticket, since I could find anything similar. Very interested to
> know if there are plans to address this (given how Samza's changelog topic is
> meant to be a persistent large state use case).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
