[
https://issues.apache.org/jira/browse/HBASE-18203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16046027#comment-16046027
]
Andrew Purtell commented on HBASE-18203:
----------------------------------------
[~Apache9] I also argue that the number of file handles should be set to an
essentially unlimited value, but there seem to be a nonzero number of
deployments that run with something more like 64k. I think it is old advice.
(We were doing this, have since raised the limit to 256k)
> Intelligently manage a pool of open references to store files
> -------------------------------------------------------------
>
> Key: HBASE-18203
> URL: https://issues.apache.org/jira/browse/HBASE-18203
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Affects Versions: 2.0.0
> Reporter: Andrew Purtell
>
> When bringing a region online we open every store file and keep the file
> open, to avoid further round trips to the HDFS namenode during reads. Naively
> keeping open every store file we encounter is a bad idea. There should be an
> upper bound. We should close and reopen files as needed once we are above the
> upper bound. We should choose candidates to close on a LRU basis. Otherwise
> we can (and some users have in production) overrun high (~64k) open file
> handle limits on the server if the aggregate number of store files is too
> large.
> Note the 'open files' here refers to open/active references to files at the
> HDFS level. How this maps to active file descriptors at the OS level depends
> on concurrency of access (block transfers, short circuit reads). The more
> open files we have at the HDFS level the higher number of OS level file
> handles we can expect to consume.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
