[
https://issues.apache.org/jira/browse/HDFS-5341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qus-jiawei updated HDFS-5341:
-----------------------------
Description:
When DirectoryScanner doing the scan function, it holding the dataset to diff
the block info between memory and disk.But it do a lot of disk operation
because it call the file's getlength funciton.
So,such disk operation should move to the diskreport.
I don't have an commit acount.So I post the patch here.
163d162
< private final long blockFileLength;
174,179d172
< if( this.blockFile != null ){
< this.blockFileLength = this.blockFile.length();
< }
< else{
< this.blockFileLength = 0;
< }
190,193d182
<
< long getBlockFileLength(){
< return blockFileLength;
< }
249,256c238
< new Daemon.DaemonFactory(){
< @Override
< public Thread newThread(Runnable
runnable) {
< Thread
t=super.newThread(runnable);
<
t.setPriority(Thread.NORM_PRIORITY);
< return t;
< }
< });
---
> new Daemon.DaemonFactory());
358d339
< LOG.info("UCADD check and update finish");
368d348
< long begin = System.currentTimeMillis();
370c350
< LOG.info("UCADD finish diskReport
using:"+(System.currentTimeMillis()-begin)+"ms");
---
>
373,375d352
< begin = System.currentTimeMillis();
< int diskHit = 0;
< LOG.info("UCADD begin to synchronized");
415,423c392,396
< || info.getBlockFileLength() !=
memBlock.getNumBytes() ) {
< //double check the block file length
< diskHit++;
< if(info.getBlockFile().length() !=
memBlock.getNumBytes()){
< // Block metadata file is missing or has
wrong generation stamp,
< // or block file length is different than
expected
< statsRecord.mismatchBlocks++;
< addDifference(diffRecord, statsRecord,
info);
< }
---
> || info.getBlockFile().length() !=
memBlock.getNumBytes()) {
> // Block metadata file is missing or has wrong
generation stamp,
> // or block file length is different than expected
> statsRecord.mismatchBlocks++;
> addDifference(diffRecord, statsRecord, info);
437d409
< LOG.info("UCADD end synchronized
using:"+(System.currentTimeMillis()-begin)+"ms diskHit:"+diskHit);
439d410
<
was:
When DirectoryScanner doing the scan function, it holding the dataset to diff
the block info between memory and disk.But it do a lot of disk operation
because it call the file's getlength funciton.
So,such disk operation should move to the diskreport.
> DirectoryScanner hold the object lock dataset too long in scan,it make the
> datanode turn into deadnode and block all reciving thread
> -------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-5341
> URL: https://issues.apache.org/jira/browse/HDFS-5341
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
> Affects Versions: 2.0.4-alpha
> Reporter: qus-jiawei
>
> When DirectoryScanner doing the scan function, it holding the dataset to diff
> the block info between memory and disk.But it do a lot of disk operation
> because it call the file's getlength funciton.
> So,such disk operation should move to the diskreport.
> I don't have an commit acount.So I post the patch here.
> 163d162
> < private final long blockFileLength;
> 174,179d172
> < if( this.blockFile != null ){
> < this.blockFileLength = this.blockFile.length();
> < }
> < else{
> < this.blockFileLength = 0;
> < }
> 190,193d182
> <
> < long getBlockFileLength(){
> < return blockFileLength;
> < }
> 249,256c238
> < new Daemon.DaemonFactory(){
> < @Override
> < public Thread newThread(Runnable
> runnable) {
> < Thread
> t=super.newThread(runnable);
> <
> t.setPriority(Thread.NORM_PRIORITY);
> < return t;
> < }
> < });
> ---
> > new Daemon.DaemonFactory());
> 358d339
> < LOG.info("UCADD check and update finish");
> 368d348
> < long begin = System.currentTimeMillis();
> 370c350
> < LOG.info("UCADD finish diskReport
> using:"+(System.currentTimeMillis()-begin)+"ms");
> ---
> >
> 373,375d352
> < begin = System.currentTimeMillis();
> < int diskHit = 0;
> < LOG.info("UCADD begin to synchronized");
> 415,423c392,396
> < || info.getBlockFileLength() !=
> memBlock.getNumBytes() ) {
> < //double check the block file length
> < diskHit++;
> < if(info.getBlockFile().length() !=
> memBlock.getNumBytes()){
> < // Block metadata file is missing or has
> wrong generation stamp,
> < // or block file length is different than
> expected
> < statsRecord.mismatchBlocks++;
> < addDifference(diffRecord, statsRecord,
> info);
> < }
> ---
> > || info.getBlockFile().length() !=
> memBlock.getNumBytes()) {
> > // Block metadata file is missing or has wrong
> generation stamp,
> > // or block file length is different than expected
> > statsRecord.mismatchBlocks++;
> > addDifference(diffRecord, statsRecord, info);
> 437d409
> < LOG.info("UCADD end synchronized
> using:"+(System.currentTimeMillis()-begin)+"ms diskHit:"+diskHit);
> 439d410
> <
--
This message was sent by Atlassian JIRA
(v6.1#6144)
