shenxianqiang created HDFS-10695:
------------------------------------
Summary: When using NN Fedration,DirectoryScanner throw
IllegalStateException
Key: HDFS-10695
URL: https://issues.apache.org/jira/browse/HDFS-10695
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Environment: nn fedration
cdh5.7.0
Reporter: shenxianqiang
DataNode DirectoryScanner always throw IllegalStateException:
{quote}
2016-07-27 10:31:58,771 ERROR
org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: Error compiling report
java.util.concurrent.ExecutionException: java.lang.IllegalStateException:
StopWatch is already running
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
at
org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.getDiskReport(DirectoryScanner.java:731)
at
org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.scan(DirectoryScanner.java:581)
at
org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.reconcile(DirectoryScanner.java:562)
at
org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.run(DirectoryScanner.java:507)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: StopWatch is already running
at org.apache.hadoop.util.StopWatch.start(StopWatch.java:49)
at
org.apache.hadoop.hdfs.server.datanode.DirectoryScanner$ReportCompiler.call(DirectoryScanner.java:812)
at
org.apache.hadoop.hdfs.server.datanode.DirectoryScanner$ReportCompiler.call(DirectoryScanner.java:778)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
... 3 more
2016-07-27 10:31:58,773 ERROR
org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: Exception during
DirectoryScanner execution - will continue next cycle
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.IllegalStateException: StopWatch is already running
at
org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.getDiskReport(DirectoryScanner.java:741)
at
org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.scan(DirectoryScanner.java:581)
at
org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.reconcile(DirectoryScanner.java:562)
at
org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.run(DirectoryScanner.java:507)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException:
java.lang.IllegalStateException: StopWatch is already running
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
at
org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.getDiskReport(DirectoryScanner.java:731)
... 10 more
Caused by: java.lang.IllegalStateException: StopWatch is already running
at org.apache.hadoop.util.StopWatch.start(StopWatch.java:49)
at
org.apache.hadoop.hdfs.server.datanode.DirectoryScanner$ReportCompiler.call(DirectoryScanner.java:812)
at
org.apache.hadoop.hdfs.server.datanode.DirectoryScanner$ReportCompiler.call(DirectoryScanner.java:778)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
... 3 more
{quote}
In DirectoryScanner.java,
{quote}
for (String bpid : bpList) {
LinkedList<ScanInfo> report = new LinkedList<>();
File bpFinalizedDir = volume.getFinalizedDir(bpid);
perfTimer.start();
throttleTimer.start();
{quote}
Because using fedration,the size of biList is greater than 1,so
perfTimer.start() will always throw IllegalStateException
In StopWatch.java
{quote}
public StopWatch start() {
if (isStarted)
throw new IllegalStateException("StopWatch is already running");
{quote}
DirectoryScanner cannot complete,thus if there is a corrupted file, datanode
cannot report this.The consequences is very serious.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
