shenxianqiang created HDFS-10695: ------------------------------------ Summary: When using NN Fedration,DirectoryScanner throw IllegalStateException Key: HDFS-10695 URL: https://issues.apache.org/jira/browse/HDFS-10695 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.6.0 Environment: nn fedration cdh5.7.0 Reporter: shenxianqiang
DataNode DirectoryScanner always throw IllegalStateException: {quote} 2016-07-27 10:31:58,771 ERROR org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: Error compiling report java.util.concurrent.ExecutionException: java.lang.IllegalStateException: StopWatch is already running at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:188) at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.getDiskReport(DirectoryScanner.java:731) at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.scan(DirectoryScanner.java:581) at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.reconcile(DirectoryScanner.java:562) at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.run(DirectoryScanner.java:507) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IllegalStateException: StopWatch is already running at org.apache.hadoop.util.StopWatch.start(StopWatch.java:49) at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner$ReportCompiler.call(DirectoryScanner.java:812) at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner$ReportCompiler.call(DirectoryScanner.java:778) at java.util.concurrent.FutureTask.run(FutureTask.java:262) ... 3 more 2016-07-27 10:31:58,773 ERROR org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: Exception during DirectoryScanner execution - will continue next cycle java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: StopWatch is already running at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.getDiskReport(DirectoryScanner.java:741) at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.scan(DirectoryScanner.java:581) at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.reconcile(DirectoryScanner.java:562) at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.run(DirectoryScanner.java:507) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: StopWatch is already running at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:188) at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.getDiskReport(DirectoryScanner.java:731) ... 10 more Caused by: java.lang.IllegalStateException: StopWatch is already running at org.apache.hadoop.util.StopWatch.start(StopWatch.java:49) at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner$ReportCompiler.call(DirectoryScanner.java:812) at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner$ReportCompiler.call(DirectoryScanner.java:778) at java.util.concurrent.FutureTask.run(FutureTask.java:262) ... 3 more {quote} In DirectoryScanner.java, {quote} for (String bpid : bpList) { LinkedList<ScanInfo> report = new LinkedList<>(); File bpFinalizedDir = volume.getFinalizedDir(bpid); perfTimer.start(); throttleTimer.start(); {quote} Because using fedration,the size of biList is greater than 1,so perfTimer.start() will always throw IllegalStateException In StopWatch.java {quote} public StopWatch start() { if (isStarted) throw new IllegalStateException("StopWatch is already running"); {quote} DirectoryScanner cannot complete,thus if there is a corrupted file, datanode cannot report this.The consequences is very serious. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org