Andrew Purtell created HBASE-18203:
--------------------------------------
Summary: Intelligently manage a pool of open references to store
files
Key: HBASE-18203
URL: https://issues.apache.org/jira/browse/HBASE-18203
Project: HBase
Issue Type: Improvement
Components: regionserver
Affects Versions: 2.0.0
Reporter: Andrew Purtell
When bringing a region online we open every store file and keep the file open,
to avoid further round trips to the HDFS namenode during reads. Naively keeping
open every store file we encounter is a bad idea. There should be an upper
bound. We should close and reopen files as needed once we are above the upper
bound. We should choose candidates to close on a LRU basis. Otherwise we can
(and some users have in production) overrun high (~64k) open file handle limits
on the server if the aggregate number of store files is too large.
Note the 'open files' here refers to open/active references to files at the
HDFS level. How this maps to active file descriptors at the OS level depends on
concurrency of access (block transfers, short circuit reads). The more open
files we have at the HDFS level the higher number of OS level file handles we
can expect to consume.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
