[ https://issues.apache.org/jira/browse/HBASE-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16177017#comment-16177017 ]
Dave Latham commented on HBASE-14247: ------------------------------------- Yes, that would address my concern, at least for our own clusters. I don't think it's a great way to address the issue, though. People with large clusters that use it would be the ones most likely to be hit by HBASE-9208. It also doesn't seem great for HBase to need to support two different directory layouts for WAL files. I still think it would be better to update the log cleaner to handle multiple directories well, and then turn it on for everyone. > Separate the old WALs into different regionserver directories > ------------------------------------------------------------- > > Key: HBASE-14247 > URL: https://issues.apache.org/jira/browse/HBASE-14247 > Project: HBase > Issue Type: Improvement > Components: wal > Reporter: Liu Shaohui > Assignee: Guanghao Zhang > Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-14247.master.001.patch, > HBASE-14247.master.002.patch, HBASE-14247-v001.diff, HBASE-14247-v002.diff, > HBASE-14247-v003.diff > > > Currently all old WALs of regionservers are achieved into the single > directory of oldWALs. In big clusters, because of long TTL of WAL or disabled > replications, the number of files under oldWALs may reach the > max-directory-items limit of HDFS, which will make the hbase cluster crashed. > {quote} > Caused by: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException): > The directory item limit of /hbase/lgprc-xiaomi/.oldlogs is exceeded: > limit=1048576 items=1048576 > {quote} > A simple solution is to separate the old WALs into different directories > according to the server name of the WAL. > Suggestions are welcomed~ Thanks -- This message was sent by Atlassian JIRA (v6.4.14#64029)