Vinayakumar B created HDFS-6114:
-----------------------------------
Summary: Block Scan log rolling will never happen if blocks
written continuously leading to huge size of dncp_block_verification.log.curr
Key: HDFS-6114
URL: https://issues.apache.org/jira/browse/HDFS-6114
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.3.0
Reporter: Vinayakumar B
Assignee: Vinayakumar B
Priority: Critical
1. {{BlockPoolSliceScanner#scan()}} will not return until all the blocks are
scanned.
2. If the blocks (with size in several MBs) to datanode are written
continuously
then one iteration of {{BlockPoolSliceScanner#scan()}} will be continously
scanning the blocks
3. These blocks will be deleted after some time (enough to get block scanned)
4. As Block Scanning is throttled, So verification of all blocks will take so
much time.
5. Rolling will never happen, so even though the total number of blocks in
datanode doesn't increases, entries ( which contains stale entries of deleted
blocks) in *dncp_block_verification.log.curr* continuously increases leading to
huge size.
In one of our env, it grown more than 1TB where total number of blocks were
only ~45k.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
