hfutatzhanghb commented on PR #5593: URL: https://github.com/apache/hadoop/pull/5593#issuecomment-1535884885
> Thanks @hfutatzhanghb for your report. > > Small dfs.namenode.replication.work.multiplier.per.iteration or big dfs.namenode.reconstruction.pending.timeout-sec may solve your problem. @ZanderXu thanks for your reply. yes, the two configuration entry may ease this problem, but still have some other problems. 1. higer data loss risky. when we make dfs.namenode.reconstruction.pending.timeout-sec big, that means in this time period, we do not care about whether the replication task is successful or not. for example, we set pending timeout 2 hours and we have a block only with one replicas left, So, namenode generates replication task, but the target datanode was dead. we have to run the risk of losing our data in this 2 hours. 2. tuning dfs.namenode.replication.work.multiplier.per.iteration small is not effective when directory is quite large. because namenode generates livenodes * multiplier blocks replication tasks every 3s. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
