[
https://issues.apache.org/jira/browse/HBASE-23598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhuobin zheng updated HBASE-23598:
----------------------------------
Attachment: HBASE-23598.patch
> There are too much small WAL File
> ---------------------------------
>
> Key: HBASE-23598
> URL: https://issues.apache.org/jira/browse/HBASE-23598
> Project: HBase
> Issue Type: Improvement
> Components: wal
> Affects Versions: 1.3.6, 2.2.2
> Environment: hbase version: cdh5-1.2.0_5.14.4
> hbase.wal.provider: multiwal
> hbase.wal.regiongrouping.numgroups: 4
> The wals file shows 100+ wal files in wal-3 , and some of them has very small
> size
> Reporter: zhuobin zheng
> Priority: Major
> Attachments: HBASE-23598.patch, wals
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I found 10W + WAL files in my 400-scale hbase cluster. Too many WAL files
> will cause the cluster and recover very slowly when cluster crash completely
> . (In the split log step) (because too many WAL files will cause too many ZK
> requests). By default, WAL files start to roll when they reach HDFS Block
> Size (256M In My Case) * 0.95. But I found that there are many small files
> (0-100M) in the WAL directory. When I look at the code , I found that when I
> configured multiwal (I configured 4 WALs for each RS), as long as a single
> WAL file reached HDFS Block Size (256M In My Case) * 0.95, all WAL files
> would scroll, so it caused a lot of WAL small files.
> I tried to modify the code to solve the problem (making each WAL scroll
> independently). Although this change is very small, I am not sure if such a
> change will cause other problems, currently being tested ...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
