[
https://issues.apache.org/jira/browse/HDFS-10602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15401742#comment-15401742
]
Yiqun Lin commented on HDFS-10602:
----------------------------------
Thanks [~xiaochen] for the comments and thanks [~liuml07] for providing the
logs.
{quote}
The 3rd DN is excluded for the test case.
{quote}
Hi, Xiao, it seems that the 3rd DN is not excluded in the test case. The
balancer moves data always between 2 specific DNs so that the 3rd DN can't
moves data.
{quote}
It seems the code has been refactored so no getBlockList exists in trunk now
{quote}
Yes, the method getBlockList in {{Balancer.java}} has been changed, now the
method was updated in {{Dispatcher#Source}}. And I have tested in the case, the
getBlockList will return the srcBlocks in the failing case.
So the problem is mainly concentrated on why the balancer moves data always
between 2 DNs. I add some codes to print the detail info.
{code}
private PendingMove chooseNextMove() {
for (Iterator<Task> i = tasks.iterator(); i.hasNext();) {
final Task task = i.next();
final DDatanode target = task.target.getDDatanode();
final PendingMove pendingBlock = new PendingMove(this, task.target);
if (target.addPendingBlock(pendingBlock)) {
// target is not busy, so do a tentative block allocation
if (pendingBlock.chooseBlockAndProxy()) {
long blockSize = pendingBlock.reportedBlock.getNumBytes(this);
incScheduledSize(-blockSize);
task.size -= blockSize;
// Print the scheduled size for test
LOG.info("TargetNode: " + target.getDatanodeInfo().getXferPort()
+ ", bytes scheduled to move, after: " + task.size
+ ", before: " + (task.size + blockSize));
if (task.size == 0) {
LOG.info("TargetNode removed.");
i.remove();
}
LOG.info("Return pendingBlock for target node "
+ target.getDatanodeInfo().getXferPort());
return pendingBlock;
...
{code}
In here the task.size which means the bytes scheduled to move will not always
reduce to 0. And then it will return this pendingBlock and the task for next
target node will be ignored. In test, I saw that the 3rd DN in balancer is
always the second targetNode in here, and the method will just return when
deals with the first target node. These are my local logs:
{code}
2016-08-01 16:51:53,466 [pool-49-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(799)) - TargetNode: 58798, bytes scheduled to
move, after: -1067, before: -967
2016-08-01 16:51:53,466 [pool-49-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(806)) - Return pendingBlock for target node
58798
2016-08-01 16:51:53,466 [pool-50-thread-10] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741833_1009 with
size=100 from 127.0.0.1:58794:DISK to 127.0.0.1:58798:DISK through
127.0.0.1:58794
2016-08-01 16:51:53,467 [pool-49-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(799)) - TargetNode: 58798, bytes scheduled to
move, after: -1167, before: -1067
2016-08-01 16:51:53,467 [pool-49-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(806)) - Return pendingBlock for target node
58798
2016-08-01 16:51:53,467 [pool-50-thread-11] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741834_1010 with
size=100 from 127.0.0.1:58794:DISK to 127.0.0.1:58798:DISK through
127.0.0.1:58794
2016-08-01 16:51:53,468 [pool-49-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(799)) - TargetNode: 58798, bytes scheduled to
move, after: -1267, before: -1167
2016-08-01 16:51:53,468 [pool-49-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(806)) - Return pendingBlock for target node
58798
2016-08-01 16:51:53,468 [pool-50-thread-12] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741835_1011 with
size=100 from 127.0.0.1:58794:DISK to 127.0.0.1:58798:DISK through
127.0.0.1:58794
2016-08-01 16:51:53,468 [pool-49-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(799)) - TargetNode: 58798, bytes scheduled to
move, after: -1367, before: -1267
2016-08-01 16:51:53,468 [pool-49-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(806)) - Return pendingBlock for target node
58798
2016-08-01 16:51:53,469 [pool-50-thread-13] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741836_1012 with
size=100 from 127.0.0.1:58794:DISK to 127.0.0.1:58798:DISK through
127.0.0.1:58794
2016-08-01 16:51:53,469 [pool-49-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(799)) - TargetNode: 58798, bytes scheduled to
move, after: -1467, before: -1367
2016-08-01 16:51:53,469 [pool-49-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(806)) - Return pendingBlock for target node
58798
2016-08-01 16:51:53,469 [pool-50-thread-14] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741837_1013 with
size=100 from 127.0.0.1:58794:DISK to 127.0.0.1:58798:DISK through
127.0.0.1:58794
2016-08-01 16:51:53,470 [pool-50-thread-15] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741838_1014 with
size=100 from 127.0.0.1:58794:DISK to 127.0.0.1:58798:DISK through
127.0.0.1:58794
2016-08-01 16:51:53,476 [DataXceiver for client /127.0.0.1:59097 [Copying block
BP-480088863-172.17.47.165-1470041466789:blk_1073741843_1019]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-480088863-172.17.47.165-1470041466789:blk_1073741843_1019 to /127.0.0.1:59097
2016-08-01 16:51:53,477 [DataXceiver for client /127.0.0.1:59109 [Copying block
BP-480088863-172.17.47.165-1470041466789:blk_1073741844_1020]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-480088863-172.17.47.165-1470041466789:blk_1073741844_1020 to /127.0.0.1:59109
2016-08-01 16:51:53,477 [DataXceiver for client /127.0.0.1:59095 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741843_1019 from
69a2e08a-ea2d-41b1-ade8-f7c8739bd07c]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-08-01 16:51:53,477 [DataXceiver for client /127.0.0.1:59098 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741844_1020 from
69a2e08a-ea2d-41b1-ade8-f7c8739bd07c]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-08-01 16:51:53,478 [DataXceiver for client /127.0.0.1:59110 [Copying block
BP-480088863-172.17.47.165-1470041466789:blk_1073741845_1021]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-480088863-172.17.47.165-1470041466789:blk_1073741845_1021 to /127.0.0.1:59110
2016-08-01 16:51:53,478 [DataXceiver for client /127.0.0.1:59111 [Copying block
BP-480088863-172.17.47.165-1470041466789:blk_1073741846_1022]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-480088863-172.17.47.165-1470041466789:blk_1073741846_1022 to /127.0.0.1:59111
2016-08-01 16:51:53,478 [DataXceiver for client /127.0.0.1:59098 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741844_1020 from
69a2e08a-ea2d-41b1-ade8-f7c8739bd07c]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-480088863-172.17.47.165-1470041466789:blk_1073741844_1020 from
/127.0.0.1:59098, delHint=69a2e08a-ea2d-41b1-ade8-f7c8739bd07c
2016-08-01 16:51:53,478 [DataXceiver for client /127.0.0.1:59095 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741843_1019 from
69a2e08a-ea2d-41b1-ade8-f7c8739bd07c]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-480088863-172.17.47.165-1470041466789:blk_1073741843_1019 from
/127.0.0.1:59095, delHint=69a2e08a-ea2d-41b1-ade8-f7c8739bd07c
2016-08-01 16:51:53,478 [pool-50-thread-5] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741844_1020 with
size=100 from 127.0.0.1:58794:DISK to 127.0.0.1:58798:DISK through
127.0.0.1:58794
2016-08-01 16:51:53,479 [pool-50-thread-4] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741843_1019 with
size=100 from 127.0.0.1:58794:DISK to 127.0.0.1:58798:DISK through
127.0.0.1:58794
2016-08-01 16:51:53,479 [DataXceiver for client /127.0.0.1:59112 [Copying block
BP-480088863-172.17.47.165-1470041466789:blk_1073741847_1023]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-480088863-172.17.47.165-1470041466789:blk_1073741847_1023 to /127.0.0.1:59112
2016-08-01 16:51:53,479 [DataXceiver for client /127.0.0.1:59100 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741846_1022 from
69a2e08a-ea2d-41b1-ade8-f7c8739bd07c]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-08-01 16:51:53,479 [DataXceiver for client /127.0.0.1:59100 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741846_1022 from
69a2e08a-ea2d-41b1-ade8-f7c8739bd07c]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-480088863-172.17.47.165-1470041466789:blk_1073741846_1022 from
/127.0.0.1:59100, delHint=69a2e08a-ea2d-41b1-ade8-f7c8739bd07c
2016-08-01 16:51:53,479 [DataXceiver for client /127.0.0.1:59101 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741847_1023 from
69a2e08a-ea2d-41b1-ade8-f7c8739bd07c]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-08-01 16:51:53,480 [DataXceiver for client /127.0.0.1:59101 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741847_1023 from
69a2e08a-ea2d-41b1-ade8-f7c8739bd07c]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-480088863-172.17.47.165-1470041466789:blk_1073741847_1023 from
/127.0.0.1:59101, delHint=69a2e08a-ea2d-41b1-ade8-f7c8739bd07c
2016-08-01 16:51:53,478 [DataXceiver for client /127.0.0.1:59099 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741845_1021 from
69a2e08a-ea2d-41b1-ade8-f7c8739bd07c]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-08-01 16:51:53,480 [DataXceiver for client /127.0.0.1:59099 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741845_1021 from
69a2e08a-ea2d-41b1-ade8-f7c8739bd07c]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-480088863-172.17.47.165-1470041466789:blk_1073741845_1021 from
/127.0.0.1:59099, delHint=69a2e08a-ea2d-41b1-ade8-f7c8739bd07c
2016-08-01 16:51:53,481 [pool-50-thread-7] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741846_1022 with
size=100 from 127.0.0.1:58794:DISK to 127.0.0.1:58798:DISK through
127.0.0.1:58794
2016-08-01 16:51:53,481 [DataXceiver for client /127.0.0.1:59113 [Copying block
BP-480088863-172.17.47.165-1470041466789:blk_1073741848_1024]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-480088863-172.17.47.165-1470041466789:blk_1073741848_1024 to /127.0.0.1:59113
2016-08-01 16:51:53,482 [DataXceiver for client /127.0.0.1:59102 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741848_1024 from
69a2e08a-ea2d-41b1-ade8-f7c8739bd07c]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-08-01 16:51:53,481 [pool-50-thread-8] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741847_1023 with
size=100 from 127.0.0.1:58794:DISK to 127.0.0.1:58798:DISK through
127.0.0.1:58794
2016-08-01 16:51:53,483 [DataXceiver for client /127.0.0.1:59114 [Copying block
BP-480088863-172.17.47.165-1470041466789:blk_1073741833_1009]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-480088863-172.17.47.165-1470041466789:blk_1073741833_1009 to /127.0.0.1:59114
2016-08-01 16:51:53,482 [DataXceiver for client /127.0.0.1:59102 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741848_1024 from
69a2e08a-ea2d-41b1-ade8-f7c8739bd07c]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-480088863-172.17.47.165-1470041466789:blk_1073741848_1024 from
/127.0.0.1:59102, delHint=69a2e08a-ea2d-41b1-ade8-f7c8739bd07c
2016-08-01 16:51:53,483 [DataXceiver for client /127.0.0.1:59103 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741833_1009 from
69a2e08a-ea2d-41b1-ade8-f7c8739bd07c]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-08-01 16:51:53,484 [DataXceiver for client /127.0.0.1:59103 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741833_1009 from
69a2e08a-ea2d-41b1-ade8-f7c8739bd07c]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-480088863-172.17.47.165-1470041466789:blk_1073741833_1009 from
/127.0.0.1:59103, delHint=69a2e08a-ea2d-41b1-ade8-f7c8739bd07c
2016-08-01 16:51:53,481 [pool-50-thread-6] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741845_1021 with
size=100 from 127.0.0.1:58794:DISK to 127.0.0.1:58798:DISK through
127.0.0.1:58794
2016-08-01 16:51:53,484 [pool-50-thread-10] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741833_1009 with
size=100 from 127.0.0.1:58794:DISK to 127.0.0.1:58798:DISK through
127.0.0.1:58794
2016-08-01 16:51:53,484 [pool-50-thread-9] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741848_1024 with
size=100 from 127.0.0.1:58794:DISK to 127.0.0.1:58798:DISK through
127.0.0.1:58794
2016-08-01 16:51:53,485 [DataXceiver for client /127.0.0.1:59115 [Copying block
BP-480088863-172.17.47.165-1470041466789:blk_1073741834_1010]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-480088863-172.17.47.165-1470041466789:blk_1073741834_1010 to /127.0.0.1:59115
2016-08-01 16:51:53,486 [DataXceiver for client /127.0.0.1:59104 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741834_1010 from
69a2e08a-ea2d-41b1-ade8-f7c8739bd07c]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-08-01 16:51:53,486 [DataXceiver for client /127.0.0.1:59104 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741834_1010 from
69a2e08a-ea2d-41b1-ade8-f7c8739bd07c]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-480088863-172.17.47.165-1470041466789:blk_1073741834_1010 from
/127.0.0.1:59104, delHint=69a2e08a-ea2d-41b1-ade8-f7c8739bd07c
2016-08-01 16:51:53,486 [pool-50-thread-11] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741834_1010 with
size=100 from 127.0.0.1:58794:DISK to 127.0.0.1:58798:DISK through
127.0.0.1:58794
2016-08-01 16:51:53,488 [DataXceiver for client /127.0.0.1:59117 [Copying block
BP-480088863-172.17.47.165-1470041466789:blk_1073741836_1012]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-480088863-172.17.47.165-1470041466789:blk_1073741836_1012 to /127.0.0.1:59117
2016-08-01 16:51:53,488 [DataXceiver for client /127.0.0.1:59106 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741836_1012 from
69a2e08a-ea2d-41b1-ade8-f7c8739bd07c]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-08-01 16:51:53,488 [DataXceiver for client /127.0.0.1:59106 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741836_1012 from
69a2e08a-ea2d-41b1-ade8-f7c8739bd07c]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-480088863-172.17.47.165-1470041466789:blk_1073741836_1012 from
/127.0.0.1:59106, delHint=69a2e08a-ea2d-41b1-ade8-f7c8739bd07c
2016-08-01 16:51:53,489 [pool-50-thread-13] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741836_1012 with
size=100 from 127.0.0.1:58794:DISK to 127.0.0.1:58798:DISK through
127.0.0.1:58794
2016-08-01 16:51:53,489 [DataXceiver for client /127.0.0.1:59107 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741837_1013 from
69a2e08a-ea2d-41b1-ade8-f7c8739bd07c]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-08-01 16:51:53,489 [DataXceiver for client /127.0.0.1:59107 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741837_1013 from
69a2e08a-ea2d-41b1-ade8-f7c8739bd07c]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-480088863-172.17.47.165-1470041466789:blk_1073741837_1013 from
/127.0.0.1:59107, delHint=69a2e08a-ea2d-41b1-ade8-f7c8739bd07c
2016-08-01 16:51:53,489 [DataXceiver for client /127.0.0.1:59118 [Copying block
BP-480088863-172.17.47.165-1470041466789:blk_1073741837_1013]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-480088863-172.17.47.165-1470041466789:blk_1073741837_1013 to /127.0.0.1:59118
2016-08-01 16:51:53,490 [DataXceiver for client /127.0.0.1:59119 [Copying block
BP-480088863-172.17.47.165-1470041466789:blk_1073741838_1014]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-480088863-172.17.47.165-1470041466789:blk_1073741838_1014 to /127.0.0.1:59119
2016-08-01 16:51:53,490 [DataXceiver for client /127.0.0.1:59108 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741838_1014 from
69a2e08a-ea2d-41b1-ade8-f7c8739bd07c]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-08-01 16:51:53,491 [DataXceiver for client /127.0.0.1:59108 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741838_1014 from
69a2e08a-ea2d-41b1-ade8-f7c8739bd07c]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-480088863-172.17.47.165-1470041466789:blk_1073741838_1014 from
/127.0.0.1:59108, delHint=69a2e08a-ea2d-41b1-ade8-f7c8739bd07c
2016-08-01 16:51:53,491 [pool-50-thread-15] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741838_1014 with
size=100 from 127.0.0.1:58794:DISK to 127.0.0.1:58798:DISK through
127.0.0.1:58794
2016-08-01 16:51:53,491 [pool-50-thread-14] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741837_1013 with
size=100 from 127.0.0.1:58794:DISK to 127.0.0.1:58798:DISK through
127.0.0.1:58794
2016-08-01 16:51:53,496 [DataXceiver for client /127.0.0.1:59116 [Copying block
BP-480088863-172.17.47.165-1470041466789:blk_1073741835_1011]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-480088863-172.17.47.165-1470041466789:blk_1073741835_1011 to /127.0.0.1:59116
2016-08-01 16:51:53,496 [DataXceiver for client /127.0.0.1:59105 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741835_1011 from
69a2e08a-ea2d-41b1-ade8-f7c8739bd07c]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-08-01 16:51:53,496 [DataXceiver for client /127.0.0.1:59105 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741835_1011 from
69a2e08a-ea2d-41b1-ade8-f7c8739bd07c]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-480088863-172.17.47.165-1470041466789:blk_1073741835_1011 from
/127.0.0.1:59105, delHint=69a2e08a-ea2d-41b1-ade8-f7c8739bd07c
2016-08-01 16:51:53,497 [pool-50-thread-12] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741835_1011 with
size=100 from 127.0.0.1:58794:DISK to 127.0.0.1:58798:DISK through
127.0.0.1:58794
2016-8-1 16:51:54 8 14.16 KB 966 B
1.43 KB
2016-08-01 16:51:57,482 [Thread-0] INFO balancer.Balancer
(Balancer.java:getLong(231)) - dfs.balancer.movedWinWidth = 2000
(default=5400000)
2016-08-01 16:51:57,482 [Thread-0] INFO balancer.Balancer
(Balancer.java:getInt(249)) - dfs.balancer.moverThreads = 1000 (default=1000)
2016-08-01 16:51:57,482 [Thread-0] INFO balancer.Balancer
(Balancer.java:getInt(249)) - dfs.balancer.dispatcherThreads = 200 (default=200)
2016-08-01 16:51:57,483 [Thread-0] INFO balancer.Balancer
(Balancer.java:getInt(249)) - dfs.datanode.balance.max.concurrent.moves = 50
(default=50)
2016-08-01 16:51:57,483 [Thread-0] INFO balancer.Balancer
(Balancer.java:getLongBytes(240)) - dfs.balancer.getBlocks.size = 2147483648
(default=2147483648)
2016-08-01 16:51:57,483 [Thread-0] INFO balancer.Balancer
(Balancer.java:getLongBytes(240)) - dfs.balancer.getBlocks.min-block-size = 1
(default=10485760)
2016-08-01 16:51:57,483 [Thread-0] INFO balancer.Balancer
(Balancer.java:getLongBytes(240)) - dfs.balancer.max-size-to-move = 10737418240
(default=10737418240)
2016-08-01 16:51:57,483 [Thread-0] INFO balancer.Balancer
(Balancer.java:getLongBytes(240)) - dfs.blocksize = 100 (default=134217728)
2016-08-01 16:51:57,485 [Thread-0] INFO net.NetworkTopology
(NetworkTopology.java:add(435)) - Adding a new node: /rack1/127.0.0.1:58794
2016-08-01 16:51:57,485 [Thread-0] INFO net.NetworkTopology
(NetworkTopology.java:add(435)) - Adding a new node: /rack0/127.0.0.1:58824
2016-08-01 16:51:57,485 [Thread-0] INFO net.NetworkTopology
(NetworkTopology.java:add(435)) - Adding a new node: /rack1/127.0.0.1:58798
2016-08-01 16:51:57,485 [Thread-0] INFO balancer.Balancer
(Balancer.java:logUtilizationCollection(430)) - 1 over-utilized:
[127.0.0.1:58798:DISK]
2016-08-01 16:51:57,485 [Thread-0] INFO balancer.Balancer
(Balancer.java:logUtilizationCollection(430)) - 0 above-average: []
2016-08-01 16:51:57,485 [Thread-0] INFO balancer.Balancer
(Balancer.java:logUtilizationCollection(430)) - 1 below-average:
[127.0.0.1:58794:DISK]
2016-08-01 16:51:57,485 [Thread-0] INFO balancer.Balancer
(Balancer.java:logUtilizationCollection(430)) - 1 underutilized:
[127.0.0.1:58824:DISK]
2016-08-01 16:51:57,485 [Thread-0] INFO balancer.Balancer
(Balancer.java:runOneIteration(603)) - Need to move 966 B to make the cluster
balanced.
2016-08-01 16:51:57,486 [IPC Server handler 1 on 58788] INFO namenode.NameNode
(NameNodeRpcServer.java:rollingUpgrade(1265)) - rollingUpgrade QUERY
2016-08-01 16:51:57,487 [Thread-0] INFO balancer.Balancer
(Balancer.java:chooseStorageGroups(459)) - chooseStorageGroups for SAME_RACK:
overUtilized => underUtilized
2016-08-01 16:51:57,487 [Thread-0] INFO balancer.Balancer
(Balancer.java:chooseStorageGroups(467)) - chooseStorageGroups for SAME_RACK:
overUtilized => belowAvgUtilized
2016-08-01 16:51:57,487 [Thread-0] INFO balancer.Balancer
(Balancer.java:matchSourceWithTargetToMove(525)) - Decided to move 33 B bytes
from 127.0.0.1:58798:DISK to 127.0.0.1:58794:DISK
2016-08-01 16:51:57,487 [Thread-0] INFO balancer.Balancer
(Balancer.java:chooseStorageGroups(475)) - chooseStorageGroups for SAME_RACK:
underUtilized => aboveAvgUtilized
2016-08-01 16:51:57,487 [Thread-0] INFO balancer.Balancer
(Balancer.java:chooseStorageGroups(459)) - chooseStorageGroups for ANY_OTHER:
overUtilized => underUtilized
2016-08-01 16:51:57,487 [Thread-0] INFO balancer.Balancer
(Balancer.java:matchSourceWithTargetToMove(525)) - Decided to move 1.40 KB
bytes from 127.0.0.1:58798:DISK to 127.0.0.1:58824:DISK
2016-08-01 16:51:57,487 [Thread-0] INFO balancer.Balancer
(Balancer.java:chooseStorageGroups(467)) - chooseStorageGroups for ANY_OTHER:
overUtilized => belowAvgUtilized
2016-08-01 16:51:57,487 [Thread-0] INFO balancer.Balancer
(Balancer.java:chooseStorageGroups(475)) - chooseStorageGroups for ANY_OTHER:
underUtilized => aboveAvgUtilized
2016-08-01 16:51:57,487 [Thread-0] INFO balancer.Balancer
(Balancer.java:runOneIteration(627)) - Will move 1.43 KB in this iteration
2016-08-01 16:51:57,488 [pool-51-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:dispatchBlocks(862)) - SrcBlocks is empty, but blcoksToReciive
is more than 0.
2016-08-01 16:51:57,492 [pool-51-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(799)) - TargetNode: 58794, bytes scheduled to
move, after: -67, before: 33
2016-08-01 16:51:57,492 [pool-51-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(806)) - Return pendingBlock for target node
58794
2016-08-01 16:51:57,492 [pool-51-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(799)) - TargetNode: 58794, bytes scheduled to
move, after: -167, before: -67
2016-08-01 16:51:57,492 [pool-51-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(806)) - Return pendingBlock for target node
58794
2016-08-01 16:51:57,492 [pool-52-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741841_1017 with
size=100 from 127.0.0.1:58798:DISK to 127.0.0.1:58794:DISK through
127.0.0.1:58798
2016-08-01 16:51:57,492 [pool-51-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(799)) - TargetNode: 58794, bytes scheduled to
move, after: -267, before: -167
2016-08-01 16:51:57,492 [pool-51-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(806)) - Return pendingBlock for target node
58794
2016-08-01 16:51:57,493 [pool-52-thread-2] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741842_1018 with
size=100 from 127.0.0.1:58798:DISK to 127.0.0.1:58794:DISK through
127.0.0.1:58798
2016-08-01 16:51:57,493 [pool-51-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(799)) - TargetNode: 58794, bytes scheduled to
move, after: -367, before: -267
2016-08-01 16:51:57,493 [pool-51-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(806)) - Return pendingBlock for target node
58794
2016-08-01 16:51:57,493 [pool-52-thread-3] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741843_1019 with
size=100 from 127.0.0.1:58798:DISK to 127.0.0.1:58794:DISK through
127.0.0.1:58798
2016-08-01 16:51:57,495 [DataXceiver for client /127.0.0.1:59123 [Copying block
BP-480088863-172.17.47.165-1470041466789:blk_1073741841_1017]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-480088863-172.17.47.165-1470041466789:blk_1073741841_1017 to /127.0.0.1:59123
2016-08-01 16:51:57,495 [DataXceiver for client /127.0.0.1:59120 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741841_1017 from
53522578-7a97-4701-925f-994893af49a3]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-08-01 16:51:57,495 [DataXceiver for client /127.0.0.1:59124 [Copying block
BP-480088863-172.17.47.165-1470041466789:blk_1073741842_1018]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-480088863-172.17.47.165-1470041466789:blk_1073741842_1018 to /127.0.0.1:59124
2016-08-01 16:51:57,496 [DataXceiver for client /127.0.0.1:59121 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741842_1018 from
53522578-7a97-4701-925f-994893af49a3]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-08-01 16:51:57,496 [DataXceiver for client /127.0.0.1:59125 [Copying block
BP-480088863-172.17.47.165-1470041466789:blk_1073741843_1019]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-480088863-172.17.47.165-1470041466789:blk_1073741843_1019 to /127.0.0.1:59125
2016-08-01 16:51:57,496 [DataXceiver for client /127.0.0.1:59121 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741842_1018 from
53522578-7a97-4701-925f-994893af49a3]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-480088863-172.17.47.165-1470041466789:blk_1073741842_1018 from
/127.0.0.1:59121, delHint=53522578-7a97-4701-925f-994893af49a3
2016-08-01 16:51:57,496 [DataXceiver for client /127.0.0.1:59122 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741843_1019 from
53522578-7a97-4701-925f-994893af49a3]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-08-01 16:51:57,497 [pool-51-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(799)) - TargetNode: 58794, bytes scheduled to
move, after: -467, before: -367
2016-08-01 16:51:57,497 [pool-51-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(806)) - Return pendingBlock for target node
58794
2016-08-01 16:51:57,497 [DataXceiver for client /127.0.0.1:59122 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741843_1019 from
53522578-7a97-4701-925f-994893af49a3]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-480088863-172.17.47.165-1470041466789:blk_1073741843_1019 from
/127.0.0.1:59122, delHint=53522578-7a97-4701-925f-994893af49a3
2016-08-01 16:51:57,497 [pool-52-thread-2] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741842_1018 with
size=100 from 127.0.0.1:58798:DISK to 127.0.0.1:58794:DISK through
127.0.0.1:58798
2016-08-01 16:51:57,497 [pool-52-thread-4] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741844_1020 with
size=100 from 127.0.0.1:58798:DISK to 127.0.0.1:58794:DISK through
127.0.0.1:58798
2016-08-01 16:51:57,497 [pool-51-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(799)) - TargetNode: 58794, bytes scheduled to
move, after: -567, before: -467
2016-08-01 16:51:57,497 [pool-51-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(806)) - Return pendingBlock for target node
58794
2016-08-01 16:51:57,498 [pool-52-thread-5] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741845_1021 with
size=100 from 127.0.0.1:58798:DISK to 127.0.0.1:58794:DISK through
127.0.0.1:58798
2016-08-01 16:51:57,497 [DataXceiver for client /127.0.0.1:59120 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741841_1017 from
53522578-7a97-4701-925f-994893af49a3]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-480088863-172.17.47.165-1470041466789:blk_1073741841_1017 from
/127.0.0.1:59120, delHint=53522578-7a97-4701-925f-994893af49a3
2016-08-01 16:51:57,497 [pool-52-thread-3] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741843_1019 with
size=100 from 127.0.0.1:58798:DISK to 127.0.0.1:58794:DISK through
127.0.0.1:58798
2016-08-01 16:51:57,498 [pool-52-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741841_1017 with
size=100 from 127.0.0.1:58798:DISK to 127.0.0.1:58794:DISK through
127.0.0.1:58798
2016-08-01 16:51:57,499 [pool-51-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(799)) - TargetNode: 58794, bytes scheduled to
move, after: -667, before: -567
2016-08-01 16:51:57,499 [pool-51-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(806)) - Return pendingBlock for target node
58794
2016-08-01 16:51:57,499 [pool-52-thread-6] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741846_1022 with
size=100 from 127.0.0.1:58798:DISK to 127.0.0.1:58794:DISK through
127.0.0.1:58798
2016-08-01 16:51:57,500 [pool-51-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(799)) - TargetNode: 58794, bytes scheduled to
move, after: -767, before: -667
2016-08-01 16:51:57,500 [pool-51-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(806)) - Return pendingBlock for target node
58794
2016-08-01 16:51:57,500 [pool-52-thread-7] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741847_1023 with
size=100 from 127.0.0.1:58798:DISK to 127.0.0.1:58794:DISK through
127.0.0.1:58798
2016-08-01 16:51:57,500 [pool-51-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(799)) - TargetNode: 58794, bytes scheduled to
move, after: -867, before: -767
2016-08-01 16:51:57,500 [pool-51-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(806)) - Return pendingBlock for target node
58794
2016-08-01 16:51:57,501 [pool-52-thread-8] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741848_1024 with
size=100 from 127.0.0.1:58798:DISK to 127.0.0.1:58794:DISK through
127.0.0.1:58798
2016-08-01 16:51:57,501 [pool-51-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(799)) - TargetNode: 58794, bytes scheduled to
move, after: -967, before: -867
2016-08-01 16:51:57,501 [pool-51-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(806)) - Return pendingBlock for target node
58794
2016-08-01 16:51:57,501 [pool-52-thread-9] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741849_1025 with
size=100 from 127.0.0.1:58798:DISK to 127.0.0.1:58794:DISK through
127.0.0.1:58798
2016-08-01 16:51:57,506 [pool-51-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(799)) - TargetNode: 58794, bytes scheduled to
move, after: -1067, before: -967
2016-08-01 16:51:57,506 [pool-52-thread-10] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741850_1026 with
size=100 from 127.0.0.1:58798:DISK to 127.0.0.1:58794:DISK through
127.0.0.1:58798
2016-08-01 16:51:57,506 [pool-51-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(806)) - Return pendingBlock for target node
58794
2016-08-01 16:51:57,507 [DataXceiver for client /127.0.0.1:59134 [Copying block
BP-480088863-172.17.47.165-1470041466789:blk_1073741846_1022]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-480088863-172.17.47.165-1470041466789:blk_1073741846_1022 to /127.0.0.1:59134
2016-08-01 16:51:57,507 [DataXceiver for client /127.0.0.1:59136 [Copying block
BP-480088863-172.17.47.165-1470041466789:blk_1073741849_1025]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-480088863-172.17.47.165-1470041466789:blk_1073741849_1025 to /127.0.0.1:59136
2016-08-01 16:51:57,508 [DataXceiver for client /127.0.0.1:59133 [Copying block
BP-480088863-172.17.47.165-1470041466789:blk_1073741845_1021]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-480088863-172.17.47.165-1470041466789:blk_1073741845_1021 to /127.0.0.1:59133
2016-08-01 16:51:57,509 [DataXceiver for client /127.0.0.1:59132 [Copying block
BP-480088863-172.17.47.165-1470041466789:blk_1073741844_1020]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-480088863-172.17.47.165-1470041466789:blk_1073741844_1020 to /127.0.0.1:59132
2016-08-01 16:51:57,509 [DataXceiver for client /127.0.0.1:59137 [Copying block
BP-480088863-172.17.47.165-1470041466789:blk_1073741848_1024]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-480088863-172.17.47.165-1470041466789:blk_1073741848_1024 to /127.0.0.1:59137
2016-08-01 16:51:57,510 [pool-51-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(799)) - TargetNode: 58794, bytes scheduled to
move, after: -1167, before: -1067
2016-08-01 16:51:57,510 [pool-51-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(806)) - Return pendingBlock for target node
58794
2016-08-01 16:51:57,510 [pool-52-thread-11] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741851_1027 with
size=100 from 127.0.0.1:58798:DISK to 127.0.0.1:58794:DISK through
127.0.0.1:58798
2016-08-01 16:51:57,510 [DataXceiver for client /127.0.0.1:59135 [Copying block
BP-480088863-172.17.47.165-1470041466789:blk_1073741847_1023]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-480088863-172.17.47.165-1470041466789:blk_1073741847_1023 to /127.0.0.1:59135
2016-08-01 16:51:57,511 [pool-51-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(799)) - TargetNode: 58794, bytes scheduled to
move, after: -1267, before: -1167
2016-08-01 16:51:57,511 [pool-51-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(806)) - Return pendingBlock for target node
58794
2016-08-01 16:51:57,511 [pool-52-thread-12] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741852_1028 with
size=100 from 127.0.0.1:58798:DISK to 127.0.0.1:58794:DISK through
127.0.0.1:58798
2016-08-01 16:51:57,512 [DataXceiver for client /127.0.0.1:59128 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741846_1022 from
53522578-7a97-4701-925f-994893af49a3]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-08-01 16:51:57,512 [DataXceiver for client /127.0.0.1:59128 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741846_1022 from
53522578-7a97-4701-925f-994893af49a3]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-480088863-172.17.47.165-1470041466789:blk_1073741846_1022 from
/127.0.0.1:59128, delHint=53522578-7a97-4701-925f-994893af49a3
2016-08-01 16:51:57,513 [DataXceiver for client /127.0.0.1:59127 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741845_1021 from
53522578-7a97-4701-925f-994893af49a3]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-08-01 16:51:57,513 [DataXceiver for client /127.0.0.1:59127 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741845_1021 from
53522578-7a97-4701-925f-994893af49a3]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-480088863-172.17.47.165-1470041466789:blk_1073741845_1021 from
/127.0.0.1:59127, delHint=53522578-7a97-4701-925f-994893af49a3
2016-08-01 16:51:57,514 [DataXceiver for client /127.0.0.1:59126 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741844_1020 from
53522578-7a97-4701-925f-994893af49a3]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-08-01 16:51:57,515 [DataXceiver for client /127.0.0.1:59126 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741844_1020 from
53522578-7a97-4701-925f-994893af49a3]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-480088863-172.17.47.165-1470041466789:blk_1073741844_1020 from
/127.0.0.1:59126, delHint=53522578-7a97-4701-925f-994893af49a3
2016-08-01 16:51:57,515 [pool-52-thread-5] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741845_1021 with
size=100 from 127.0.0.1:58798:DISK to 127.0.0.1:58794:DISK through
127.0.0.1:58798
2016-08-01 16:51:57,515 [pool-52-thread-6] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741846_1022 with
size=100 from 127.0.0.1:58798:DISK to 127.0.0.1:58794:DISK through
127.0.0.1:58798
2016-08-01 16:51:57,516 [pool-51-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(799)) - TargetNode: 58794, bytes scheduled to
move, after: -1367, before: -1267
2016-08-01 16:51:57,516 [pool-51-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(806)) - Return pendingBlock for target node
58794
2016-08-01 16:51:57,516 [DataXceiver for client /127.0.0.1:59129 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741847_1023 from
53522578-7a97-4701-925f-994893af49a3]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-08-01 16:51:57,516 [pool-52-thread-4] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741844_1020 with
size=100 from 127.0.0.1:58798:DISK to 127.0.0.1:58794:DISK through
127.0.0.1:58798
2016-08-01 16:51:57,516 [DataXceiver for client /127.0.0.1:59129 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741847_1023 from
53522578-7a97-4701-925f-994893af49a3]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-480088863-172.17.47.165-1470041466789:blk_1073741847_1023 from
/127.0.0.1:59129, delHint=53522578-7a97-4701-925f-994893af49a3
2016-08-01 16:51:57,517 [pool-52-thread-7] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741847_1023 with
size=100 from 127.0.0.1:58798:DISK to 127.0.0.1:58794:DISK through
127.0.0.1:58798
2016-08-01 16:51:57,517 [pool-51-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(799)) - TargetNode: 58794, bytes scheduled to
move, after: -1467, before: -1367
2016-08-01 16:51:57,517 [pool-51-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:chooseNextMove(806)) - Return pendingBlock for target node
58794
2016-08-01 16:51:57,517 [DataXceiver for client /127.0.0.1:59130 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741848_1024 from
53522578-7a97-4701-925f-994893af49a3]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-08-01 16:51:57,517 [pool-52-thread-14] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741854_1030 with
size=100 from 127.0.0.1:58798:DISK to 127.0.0.1:58794:DISK through
127.0.0.1:58798
2016-08-01 16:51:57,518 [DataXceiver for client /127.0.0.1:59130 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741848_1024 from
53522578-7a97-4701-925f-994893af49a3]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-480088863-172.17.47.165-1470041466789:blk_1073741848_1024 from
/127.0.0.1:59130, delHint=53522578-7a97-4701-925f-994893af49a3
2016-08-01 16:51:57,518 [pool-52-thread-13] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741853_1029 with
size=100 from 127.0.0.1:58798:DISK to 127.0.0.1:58794:DISK through
127.0.0.1:58798
2016-08-01 16:51:57,518 [DataXceiver for client /127.0.0.1:59131 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741849_1025 from
53522578-7a97-4701-925f-994893af49a3]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-08-01 16:51:57,518 [DataXceiver for client /127.0.0.1:59131 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741849_1025 from
53522578-7a97-4701-925f-994893af49a3]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-480088863-172.17.47.165-1470041466789:blk_1073741849_1025 from
/127.0.0.1:59131, delHint=53522578-7a97-4701-925f-994893af49a3
2016-08-01 16:51:57,519 [pool-52-thread-8] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741848_1024 with
size=100 from 127.0.0.1:58798:DISK to 127.0.0.1:58794:DISK through
127.0.0.1:58798
2016-08-01 16:51:57,519 [pool-52-thread-9] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741849_1025 with
size=100 from 127.0.0.1:58798:DISK to 127.0.0.1:58794:DISK through
127.0.0.1:58798
2016-08-01 16:51:57,520 [pool-52-thread-15] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741825_1001 with
size=100 from 127.0.0.1:58798:DISK to 127.0.0.1:58794:DISK through
127.0.0.1:58798
2016-08-01 16:51:57,521 [DataXceiver for client /127.0.0.1:59140 [Copying block
BP-480088863-172.17.47.165-1470041466789:blk_1073741850_1026]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-480088863-172.17.47.165-1470041466789:blk_1073741850_1026 to /127.0.0.1:59140
2016-08-01 16:51:57,522 [DataXceiver for client /127.0.0.1:59138 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741850_1026 from
53522578-7a97-4701-925f-994893af49a3]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-08-01 16:51:57,522 [DataXceiver for client /127.0.0.1:59138 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741850_1026 from
53522578-7a97-4701-925f-994893af49a3]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-480088863-172.17.47.165-1470041466789:blk_1073741850_1026 from
/127.0.0.1:59138, delHint=53522578-7a97-4701-925f-994893af49a3
2016-08-01 16:51:57,523 [pool-52-thread-10] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741850_1026 with
size=100 from 127.0.0.1:58798:DISK to 127.0.0.1:58794:DISK through
127.0.0.1:58798
2016-08-01 16:51:57,524 [DataXceiver for client /127.0.0.1:59141 [Copying block
BP-480088863-172.17.47.165-1470041466789:blk_1073741851_1027]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-480088863-172.17.47.165-1470041466789:blk_1073741851_1027 to /127.0.0.1:59141
2016-08-01 16:51:57,525 [DataXceiver for client /127.0.0.1:59143 [Copying block
BP-480088863-172.17.47.165-1470041466789:blk_1073741852_1028]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-480088863-172.17.47.165-1470041466789:blk_1073741852_1028 to /127.0.0.1:59143
2016-08-01 16:51:57,525 [DataXceiver for client /127.0.0.1:59139 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741851_1027 from
53522578-7a97-4701-925f-994893af49a3]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-08-01 16:51:57,525 [DataXceiver for client /127.0.0.1:59142 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741852_1028 from
53522578-7a97-4701-925f-994893af49a3]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-08-01 16:51:57,525 [DataXceiver for client /127.0.0.1:59139 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741851_1027 from
53522578-7a97-4701-925f-994893af49a3]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-480088863-172.17.47.165-1470041466789:blk_1073741851_1027 from
/127.0.0.1:59139, delHint=53522578-7a97-4701-925f-994893af49a3
2016-08-01 16:51:57,525 [DataXceiver for client /127.0.0.1:59142 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741852_1028 from
53522578-7a97-4701-925f-994893af49a3]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-480088863-172.17.47.165-1470041466789:blk_1073741852_1028 from
/127.0.0.1:59142, delHint=53522578-7a97-4701-925f-994893af49a3
2016-08-01 16:51:57,526 [pool-52-thread-11] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741851_1027 with
size=100 from 127.0.0.1:58798:DISK to 127.0.0.1:58794:DISK through
127.0.0.1:58798
2016-08-01 16:51:57,526 [pool-52-thread-12] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741852_1028 with
size=100 from 127.0.0.1:58798:DISK to 127.0.0.1:58794:DISK through
127.0.0.1:58798
2016-08-01 16:51:57,528 [DataXceiver for client /127.0.0.1:59144 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741854_1030 from
53522578-7a97-4701-925f-994893af49a3]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-08-01 16:51:57,529 [DataXceiver for client /127.0.0.1:59145 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741853_1029 from
53522578-7a97-4701-925f-994893af49a3]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-08-01 16:51:57,529 [DataXceiver for client /127.0.0.1:59144 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741854_1030 from
53522578-7a97-4701-925f-994893af49a3]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-480088863-172.17.47.165-1470041466789:blk_1073741854_1030 from
/127.0.0.1:59144, delHint=53522578-7a97-4701-925f-994893af49a3
2016-08-01 16:51:57,529 [DataXceiver for client /127.0.0.1:59145 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741853_1029 from
53522578-7a97-4701-925f-994893af49a3]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-480088863-172.17.47.165-1470041466789:blk_1073741853_1029 from
/127.0.0.1:59145, delHint=53522578-7a97-4701-925f-994893af49a3
2016-08-01 16:51:57,528 [DataXceiver for client /127.0.0.1:59147 [Copying block
BP-480088863-172.17.47.165-1470041466789:blk_1073741854_1030]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-480088863-172.17.47.165-1470041466789:blk_1073741854_1030 to /127.0.0.1:59147
2016-08-01 16:51:57,530 [DataXceiver for client /127.0.0.1:59146 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741825_1001 from
53522578-7a97-4701-925f-994893af49a3]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-08-01 16:51:57,530 [DataXceiver for client /127.0.0.1:59146 [Replacing
block BP-480088863-172.17.47.165-1470041466789:blk_1073741825_1001 from
53522578-7a97-4701-925f-994893af49a3]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-480088863-172.17.47.165-1470041466789:blk_1073741825_1001 from
/127.0.0.1:59146, delHint=53522578-7a97-4701-925f-994893af49a3
2016-08-01 16:51:57,528 [DataXceiver for client /127.0.0.1:59148 [Copying block
BP-480088863-172.17.47.165-1470041466789:blk_1073741853_1029]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-480088863-172.17.47.165-1470041466789:blk_1073741853_1029 to /127.0.0.1:59148
2016-08-01 16:51:57,529 [DataXceiver for client /127.0.0.1:59149 [Copying block
BP-480088863-172.17.47.165-1470041466789:blk_1073741825_1001]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-480088863-172.17.47.165-1470041466789:blk_1073741825_1001 to /127.0.0.1:59149
2016-08-01 16:51:57,531 [pool-52-thread-14] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741854_1030 with
size=100 from 127.0.0.1:58798:DISK to 127.0.0.1:58794:DISK through
127.0.0.1:58798
2016-08-01 16:51:57,532 [pool-52-thread-13] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741853_1029 with
size=100 from 127.0.0.1:58798:DISK to 127.0.0.1:58794:DISK through
127.0.0.1:58798
2016-08-01 16:51:57,533 [pool-52-thread-15] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741825_1001 with
size=100 from 127.0.0.1:58798:DISK to 127.0.0.1:58794:DISK through
127.0.0.1:58798
2016-8-1 16:51:58 9 15.63 KB 966 B
1.43 KB
{code}
So I think it seems a bug in choosing pending blocks.
{code}
long blockSize = pendingBlock.reportedBlock.getNumBytes(this);
incScheduledSize(-blockSize);
task.size -= blockSize;
// here should be task.size <= 0
if (task.size == 0) {
LOG.info("TargetNode removed.");
i.remove();
}
return pendingBlock;
...
{code}
The task should be removed when the task.size was first reduced to 0 or less
than 0. We can see in this failling case, the balancer will not finished and it
always balancer between twn DNs.
Finally, post a new patch for this. I have tested in my local, the result is
good.
> TestBalancer runs timeout intermittently
> ----------------------------------------
>
> Key: HDFS-10602
> URL: https://issues.apache.org/jira/browse/HDFS-10602
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 3.0.0-alpha1
> Reporter: Yiqun Lin
> Assignee: Yiqun Lin
> Attachments: HDFS-10602.001.patch, fail.log, pass.log
>
>
> As the jira HDFS-10336 has mentioned, the unit test
> {{TestBalancer#testBalancerWithKeytabs}} will runs too slowly sometimes and
> that leads the timeout. The test {{TestBalancer#testUnknownDatanodeSimple}}
> will also has this problem. These two tests both use the method
> {{testUnknownDatanode}}. We can do some optimization for this method.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
