lei w created HDFS-16058:
----------------------------
Summary: If the balancer cannot update the datanode's storage type
in time, the balancer will throw NullPointerException
Key: HDFS-16058
URL: https://issues.apache.org/jira/browse/HDFS-16058
Project: Hadoop HDFS
Issue Type: Bug
Components: balancer & mover
Reporter: lei w
The main logic of the balancer is to initialize the cluster information (number
of datanodes, datanode's storage type and network topology etc)ćcalculate the
source DataNode and target DataNode that need to move the block and then obtain
the movable block through the getblocks(). Finally , move blocks. If a DataNode
adds another type of storage and writes an EC block after the cluster
information is initialized, and then returns the newly written EC block to
balancer by getblocks(), it will cause balancer to throw NullPointerException
.The main reason is that the balancer does not record the newly added storage
type of the DataNode, which causes the current DataNode information not to be
recorded when generate an object of type DBlockStriped . Finally, null is
returned when the block of the current DataNode is obtained through the object
of type DBlockStriped.
{code:java}
2021-04-23 19:38:21,233 WARN org.apache.hadoop.hdfs.server.balancer.Dispatcher:
Dispatcher thread failed
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.chooseProxySource(Dispatcher.java:325)
at
org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.markMovedIfGoodBlock(Dispatcher.java:291)
at
org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.chooseBlockAndProxy(Dispatcher.java:271)
at
org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.access$2500(Dispatcher.java:235)
at
org.apache.hadoop.hdfs.server.balancer.Dispatcher$Source.chooseNextMove(Dispatcher.java:886)
at
org.apache.hadoop.hdfs.server.balancer.Dispatcher$Source.dispatchBlocks(Dispatcher.java:943)
at
org.apache.hadoop.hdfs.server.balancer.Dispatcher$Source.access$3200(Dispatcher.java:751)
at
org.apache.hadoop.hdfs.server.balancer.Dispatcher$2.run(Dispatcher.java:1221)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
