[
https://issues.apache.org/jira/browse/HDFS-10602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399300#comment-15399300
]
Yiqun Lin commented on HDFS-10602:
----------------------------------
The test {{TestBalancer#testUnknownDatanodeSimple}} failed again in the recent
jenkin building(Like in HDFS-10625), and all of the reason are timed out. But I
tested this many times in my local, all of the test result were passed and it
only costed 14~19 seconds. Then I made some change in the method
{{testUnknownDatanode}} to delay the balancer operations:
{code}
//...
cluster.startDataNodes(conf, 1, true, null,
new String[]{RACK0}, null,new long[]{CAPACITY});
cluster.triggerHeartbeats();
// Sleep sometime for test
Thread.sleep(9000);
Collection<URI> namenodes = DFSUtil.getInternalNsRpcUris(conf);
Set<String> datanodes = new HashSet<String>();
datanodes.add(cluster.getDataNodes().get(0).getDatanodeId().getHostName());
BalancerParameters.Builder pBuilder =
new BalancerParameters.Builder();
pBuilder.setExcludedNodes(datanodes);
pBuilder.setRunDuringUpgrade(false);
final int r = Balancer.run(namenodes, pBuilder.build(), conf);
assertEquals(ExitStatus.SUCCESS.getExitCode(), r);
{code}
Because sometimes Namenode might process block report slowly, so I made this
change to simulate this scenario. Based on this change, the test executed
timedout intermittently. From the output logs, I found the balancer in the test
will not be finished. The value for {{bytesLeftToMove}} was always be same. The
logs in my local env:
{code}
2016-7-29 20:15:24 3 5.18 KB 533 B
1.01 KB
2016-07-29 20:15:27,739 [Thread-0] INFO balancer.Balancer
(Balancer.java:getLong(231)) - dfs.balancer.movedWinWidth = 2000
(default=5400000)
2016-07-29 20:15:27,740 [Thread-0] INFO balancer.Balancer
(Balancer.java:getInt(249)) - dfs.balancer.moverThreads = 1000 (default=1000)
2016-07-29 20:15:27,740 [Thread-0] INFO balancer.Balancer
(Balancer.java:getInt(249)) - dfs.balancer.dispatcherThreads = 200 (default=200)
2016-07-29 20:15:27,740 [Thread-0] INFO balancer.Balancer
(Balancer.java:getInt(249)) - dfs.datanode.balance.max.concurrent.moves = 50
(default=50)
2016-07-29 20:15:27,740 [Thread-0] INFO balancer.Balancer
(Balancer.java:getLongBytes(240)) - dfs.balancer.getBlocks.size = 2147483648
(default=2147483648)
2016-07-29 20:15:27,740 [Thread-0] INFO balancer.Balancer
(Balancer.java:getLongBytes(240)) - dfs.balancer.getBlocks.min-block-size = 1
(default=10485760)
2016-07-29 20:15:27,740 [Thread-0] INFO balancer.Balancer
(Balancer.java:getLongBytes(240)) - dfs.balancer.max-size-to-move = 10737418240
(default=10737418240)
2016-07-29 20:15:27,740 [Thread-0] INFO balancer.Balancer
(Balancer.java:getLongBytes(240)) - dfs.blocksize = 100 (default=134217728)
2016-07-29 20:15:27,741 [Thread-0] INFO net.NetworkTopology
(NetworkTopology.java:add(435)) - Adding a new node: /rack1/127.0.0.1:65164
2016-07-29 20:15:27,742 [Thread-0] INFO net.NetworkTopology
(NetworkTopology.java:add(435)) - Adding a new node: /rack0/127.0.0.1:65169
2016-07-29 20:15:27,742 [Thread-0] INFO net.NetworkTopology
(NetworkTopology.java:add(435)) - Adding a new node: /rack1/127.0.0.1:65160
2016-07-29 20:15:27,742 [Thread-0] INFO balancer.Balancer
(Balancer.java:logUtilizationCollection(430)) - 1 over-utilized:
[127.0.0.1:65160:DISK]
2016-07-29 20:15:27,742 [Thread-0] INFO balancer.Balancer
(Balancer.java:logUtilizationCollection(430)) - 0 above-average: []
2016-07-29 20:15:27,742 [Thread-0] INFO balancer.Balancer
(Balancer.java:logUtilizationCollection(430)) - 1 below-average:
[127.0.0.1:65164:DISK]
2016-07-29 20:15:27,742 [Thread-0] INFO balancer.Balancer
(Balancer.java:logUtilizationCollection(430)) - 1 underutilized:
[127.0.0.1:65169:DISK]
2016-07-29 20:15:27,742 [Thread-0] INFO balancer.Balancer
(Balancer.java:runOneIteration(603)) - Need to move 533 B to make the cluster
balanced.
2016-07-29 20:15:27,744 [IPC Server handler 6 on 65154] INFO namenode.NameNode
(NameNodeRpcServer.java:rollingUpgrade(1265)) - rollingUpgrade QUERY
2016-07-29 20:15:27,744 [Thread-0] INFO balancer.Balancer
(Balancer.java:chooseStorageGroups(459)) - chooseStorageGroups for SAME_RACK:
overUtilized => underUtilized
2016-07-29 20:15:27,744 [Thread-0] INFO balancer.Balancer
(Balancer.java:chooseStorageGroups(467)) - chooseStorageGroups for SAME_RACK:
overUtilized => belowAvgUtilized
2016-07-29 20:15:27,744 [Thread-0] INFO balancer.Balancer
(Balancer.java:matchSourceWithTargetToMove(525)) - Decided to move 66 B bytes
from 127.0.0.1:65160:DISK to 127.0.0.1:65164:DISK
2016-07-29 20:15:27,744 [Thread-0] INFO balancer.Balancer
(Balancer.java:chooseStorageGroups(475)) - chooseStorageGroups for SAME_RACK:
underUtilized => aboveAvgUtilized
2016-07-29 20:15:27,744 [Thread-0] INFO balancer.Balancer
(Balancer.java:chooseStorageGroups(459)) - chooseStorageGroups for ANY_OTHER:
overUtilized => underUtilized
2016-07-29 20:15:27,744 [Thread-0] INFO balancer.Balancer
(Balancer.java:matchSourceWithTargetToMove(525)) - Decided to move 966 B bytes
from 127.0.0.1:65160:DISK to 127.0.0.1:65169:DISK
2016-07-29 20:15:27,744 [Thread-0] INFO balancer.Balancer
(Balancer.java:chooseStorageGroups(467)) - chooseStorageGroups for ANY_OTHER:
overUtilized => belowAvgUtilized
2016-07-29 20:15:27,744 [Thread-0] INFO balancer.Balancer
(Balancer.java:chooseStorageGroups(475)) - chooseStorageGroups for ANY_OTHER:
underUtilized => aboveAvgUtilized
2016-07-29 20:15:27,744 [Thread-0] INFO balancer.Balancer
(Balancer.java:runOneIteration(627)) - Will move 1.01 KB in this iteration
2016-07-29 20:15:27,748 [pool-42-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741841_1017 with
size=100 from 127.0.0.1:65160:DISK to 127.0.0.1:65164:DISK through
127.0.0.1:65160
2016-07-29 20:15:27,748 [pool-42-thread-2] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741842_1018 with
size=100 from 127.0.0.1:65160:DISK to 127.0.0.1:65164:DISK through
127.0.0.1:65160
2016-07-29 20:15:27,748 [pool-42-thread-3] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741843_1019 with
size=100 from 127.0.0.1:65160:DISK to 127.0.0.1:65164:DISK through
127.0.0.1:65160
2016-07-29 20:15:27,748 [pool-42-thread-4] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741844_1020 with
size=100 from 127.0.0.1:65160:DISK to 127.0.0.1:65164:DISK through
127.0.0.1:65160
2016-07-29 20:15:27,749 [pool-42-thread-5] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741845_1021 with
size=100 from 127.0.0.1:65160:DISK to 127.0.0.1:65164:DISK through
127.0.0.1:65160
2016-07-29 20:15:27,749 [pool-42-thread-6] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741846_1022 with
size=100 from 127.0.0.1:65160:DISK to 127.0.0.1:65164:DISK through
127.0.0.1:65160
2016-07-29 20:15:27,752 [pool-42-thread-7] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741847_1023 with
size=100 from 127.0.0.1:65160:DISK to 127.0.0.1:65164:DISK through
127.0.0.1:65160
2016-07-29 20:15:27,752 [pool-42-thread-8] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741834_1010 with
size=100 from 127.0.0.1:65160:DISK to 127.0.0.1:65164:DISK through
127.0.0.1:65160
2016-07-29 20:15:27,753 [pool-42-thread-9] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741835_1011 with
size=100 from 127.0.0.1:65160:DISK to 127.0.0.1:65164:DISK through
127.0.0.1:65160
2016-07-29 20:15:27,753 [pool-42-thread-10] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741836_1012 with
size=100 from 127.0.0.1:65160:DISK to 127.0.0.1:65164:DISK through
127.0.0.1:65160
2016-07-29 20:15:27,753 [pool-42-thread-11] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741837_1013 with
size=100 from 127.0.0.1:65160:DISK to 127.0.0.1:65164:DISK through
127.0.0.1:65160
2016-07-29 20:15:27,755 [DataXceiver for client /127.0.0.1:65307 [Copying block
BP-2103530174-172.17.47.165-1469794498037:blk_1073741841_1017]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-2103530174-172.17.47.165-1469794498037:blk_1073741841_1017 to
/127.0.0.1:65307
2016-07-29 20:15:27,755 [DataXceiver for client /127.0.0.1:65308 [Copying block
BP-2103530174-172.17.47.165-1469794498037:blk_1073741842_1018]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-2103530174-172.17.47.165-1469794498037:blk_1073741842_1018 to
/127.0.0.1:65308
2016-07-29 20:15:27,755 [DataXceiver for client /127.0.0.1:65301 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741841_1017 from
f5e9cdf3-e21a-440b-b728-01083da1b924]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-07-29 20:15:27,756 [DataXceiver for client /127.0.0.1:65302 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741842_1018 from
f5e9cdf3-e21a-440b-b728-01083da1b924]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-07-29 20:15:27,756 [DataXceiver for client /127.0.0.1:65302 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741842_1018 from
f5e9cdf3-e21a-440b-b728-01083da1b924]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-2103530174-172.17.47.165-1469794498037:blk_1073741842_1018 from
/127.0.0.1:65302, delHint=f5e9cdf3-e21a-440b-b728-01083da1b924
2016-07-29 20:15:27,756 [DataXceiver for client /127.0.0.1:65301 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741841_1017 from
f5e9cdf3-e21a-440b-b728-01083da1b924]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-2103530174-172.17.47.165-1469794498037:blk_1073741841_1017 from
/127.0.0.1:65301, delHint=f5e9cdf3-e21a-440b-b728-01083da1b924
2016-07-29 20:15:27,757 [pool-42-thread-2] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741842_1018 with
size=100 from 127.0.0.1:65160:DISK to 127.0.0.1:65164:DISK through
127.0.0.1:65160
2016-07-29 20:15:27,757 [pool-42-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741841_1017 with
size=100 from 127.0.0.1:65160:DISK to 127.0.0.1:65164:DISK through
127.0.0.1:65160
2016-07-29 20:15:27,757 [DataXceiver for client /127.0.0.1:65309 [Copying block
BP-2103530174-172.17.47.165-1469794498037:blk_1073741843_1019]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-2103530174-172.17.47.165-1469794498037:blk_1073741843_1019 to
/127.0.0.1:65309
2016-07-29 20:15:27,758 [DataXceiver for client /127.0.0.1:65310 [Copying block
BP-2103530174-172.17.47.165-1469794498037:blk_1073741844_1020]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-2103530174-172.17.47.165-1469794498037:blk_1073741844_1020 to
/127.0.0.1:65310
2016-07-29 20:15:27,758 [DataXceiver for client /127.0.0.1:65304 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741844_1020 from
f5e9cdf3-e21a-440b-b728-01083da1b924]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-07-29 20:15:27,758 [DataXceiver for client /127.0.0.1:65304 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741844_1020 from
f5e9cdf3-e21a-440b-b728-01083da1b924]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-2103530174-172.17.47.165-1469794498037:blk_1073741844_1020 from
/127.0.0.1:65304, delHint=f5e9cdf3-e21a-440b-b728-01083da1b924
2016-07-29 20:15:27,758 [DataXceiver for client /127.0.0.1:65303 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741843_1019 from
f5e9cdf3-e21a-440b-b728-01083da1b924]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-07-29 20:15:27,759 [pool-42-thread-4] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741844_1020 with
size=100 from 127.0.0.1:65160:DISK to 127.0.0.1:65164:DISK through
127.0.0.1:65160
2016-07-29 20:15:27,759 [DataXceiver for client /127.0.0.1:65303 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741843_1019 from
f5e9cdf3-e21a-440b-b728-01083da1b924]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-2103530174-172.17.47.165-1469794498037:blk_1073741843_1019 from
/127.0.0.1:65303, delHint=f5e9cdf3-e21a-440b-b728-01083da1b924
2016-07-29 20:15:27,759 [pool-42-thread-3] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741843_1019 with
size=100 from 127.0.0.1:65160:DISK to 127.0.0.1:65164:DISK through
127.0.0.1:65160
2016-07-29 20:15:27,761 [DataXceiver for client /127.0.0.1:65311 [Copying block
BP-2103530174-172.17.47.165-1469794498037:blk_1073741845_1021]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-2103530174-172.17.47.165-1469794498037:blk_1073741845_1021 to
/127.0.0.1:65311
2016-07-29 20:15:27,761 [DataXceiver for client /127.0.0.1:65305 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741845_1021 from
f5e9cdf3-e21a-440b-b728-01083da1b924]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-07-29 20:15:27,761 [DataXceiver for client /127.0.0.1:65305 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741845_1021 from
f5e9cdf3-e21a-440b-b728-01083da1b924]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-2103530174-172.17.47.165-1469794498037:blk_1073741845_1021 from
/127.0.0.1:65305, delHint=f5e9cdf3-e21a-440b-b728-01083da1b924
2016-07-29 20:15:27,762 [pool-42-thread-5] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741845_1021 with
size=100 from 127.0.0.1:65160:DISK to 127.0.0.1:65164:DISK through
127.0.0.1:65160
2016-07-29 20:15:27,762 [DataXceiver for client /127.0.0.1:65312 [Copying block
BP-2103530174-172.17.47.165-1469794498037:blk_1073741846_1022]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-2103530174-172.17.47.165-1469794498037:blk_1073741846_1022 to
/127.0.0.1:65312
2016-07-29 20:15:27,762 [DataXceiver for client /127.0.0.1:65306 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741846_1022 from
f5e9cdf3-e21a-440b-b728-01083da1b924]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-07-29 20:15:27,763 [DataXceiver for client /127.0.0.1:65306 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741846_1022 from
f5e9cdf3-e21a-440b-b728-01083da1b924]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-2103530174-172.17.47.165-1469794498037:blk_1073741846_1022 from
/127.0.0.1:65306, delHint=f5e9cdf3-e21a-440b-b728-01083da1b924
2016-07-29 20:15:27,763 [pool-42-thread-6] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741846_1022 with
size=100 from 127.0.0.1:65160:DISK to 127.0.0.1:65164:DISK through
127.0.0.1:65160
2016-07-29 20:15:27,764 [DataXceiver for client /127.0.0.1:65318 [Copying block
BP-2103530174-172.17.47.165-1469794498037:blk_1073741847_1023]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-2103530174-172.17.47.165-1469794498037:blk_1073741847_1023 to
/127.0.0.1:65318
2016-07-29 20:15:27,765 [DataXceiver for client /127.0.0.1:65313 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741847_1023 from
f5e9cdf3-e21a-440b-b728-01083da1b924]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-07-29 20:15:27,765 [DataXceiver for client /127.0.0.1:65313 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741847_1023 from
f5e9cdf3-e21a-440b-b728-01083da1b924]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-2103530174-172.17.47.165-1469794498037:blk_1073741847_1023 from
/127.0.0.1:65313, delHint=f5e9cdf3-e21a-440b-b728-01083da1b924
2016-07-29 20:15:27,765 [DataXceiver for client /127.0.0.1:65319 [Copying block
BP-2103530174-172.17.47.165-1469794498037:blk_1073741834_1010]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-2103530174-172.17.47.165-1469794498037:blk_1073741834_1010 to
/127.0.0.1:65319
2016-07-29 20:15:27,765 [pool-42-thread-7] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741847_1023 with
size=100 from 127.0.0.1:65160:DISK to 127.0.0.1:65164:DISK through
127.0.0.1:65160
2016-07-29 20:15:27,766 [DataXceiver for client /127.0.0.1:65314 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741834_1010 from
f5e9cdf3-e21a-440b-b728-01083da1b924]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-07-29 20:15:27,766 [DataXceiver for client /127.0.0.1:65314 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741834_1010 from
f5e9cdf3-e21a-440b-b728-01083da1b924]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-2103530174-172.17.47.165-1469794498037:blk_1073741834_1010 from
/127.0.0.1:65314, delHint=f5e9cdf3-e21a-440b-b728-01083da1b924
2016-07-29 20:15:27,766 [pool-42-thread-8] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741834_1010 with
size=100 from 127.0.0.1:65160:DISK to 127.0.0.1:65164:DISK through
127.0.0.1:65160
2016-07-29 20:15:27,767 [DataXceiver for client /127.0.0.1:65315 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741835_1011 from
f5e9cdf3-e21a-440b-b728-01083da1b924]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-07-29 20:15:27,766 [DataXceiver for client /127.0.0.1:65320 [Copying block
BP-2103530174-172.17.47.165-1469794498037:blk_1073741835_1011]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-2103530174-172.17.47.165-1469794498037:blk_1073741835_1011 to
/127.0.0.1:65320
2016-07-29 20:15:27,767 [DataXceiver for client /127.0.0.1:65315 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741835_1011 from
f5e9cdf3-e21a-440b-b728-01083da1b924]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-2103530174-172.17.47.165-1469794498037:blk_1073741835_1011 from
/127.0.0.1:65315, delHint=f5e9cdf3-e21a-440b-b728-01083da1b924
2016-07-29 20:15:27,767 [pool-42-thread-9] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741835_1011 with
size=100 from 127.0.0.1:65160:DISK to 127.0.0.1:65164:DISK through
127.0.0.1:65160
2016-07-29 20:15:27,767 [DataXceiver for client /127.0.0.1:65321 [Copying block
BP-2103530174-172.17.47.165-1469794498037:blk_1073741836_1012]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-2103530174-172.17.47.165-1469794498037:blk_1073741836_1012 to
/127.0.0.1:65321
2016-07-29 20:15:27,768 [DataXceiver for client /127.0.0.1:65316 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741836_1012 from
f5e9cdf3-e21a-440b-b728-01083da1b924]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-07-29 20:15:27,768 [DataXceiver for client /127.0.0.1:65316 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741836_1012 from
f5e9cdf3-e21a-440b-b728-01083da1b924]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-2103530174-172.17.47.165-1469794498037:blk_1073741836_1012 from
/127.0.0.1:65316, delHint=f5e9cdf3-e21a-440b-b728-01083da1b924
2016-07-29 20:15:27,768 [pool-42-thread-10] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741836_1012 with
size=100 from 127.0.0.1:65160:DISK to 127.0.0.1:65164:DISK through
127.0.0.1:65160
2016-07-29 20:15:27,769 [DataXceiver for client /127.0.0.1:65322 [Copying block
BP-2103530174-172.17.47.165-1469794498037:blk_1073741837_1013]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-2103530174-172.17.47.165-1469794498037:blk_1073741837_1013 to
/127.0.0.1:65322
2016-07-29 20:15:27,769 [DataXceiver for client /127.0.0.1:65317 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741837_1013 from
f5e9cdf3-e21a-440b-b728-01083da1b924]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-07-29 20:15:27,769 [DataXceiver for client /127.0.0.1:65317 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741837_1013 from
f5e9cdf3-e21a-440b-b728-01083da1b924]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-2103530174-172.17.47.165-1469794498037:blk_1073741837_1013 from
/127.0.0.1:65317, delHint=f5e9cdf3-e21a-440b-b728-01083da1b924
2016-07-29 20:15:27,769 [pool-42-thread-11] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741837_1013 with
size=100 from 127.0.0.1:65160:DISK to 127.0.0.1:65164:DISK through
127.0.0.1:65160
2016-7-29 20:15:28 4 6.25 KB 533 B
1.01 KB
2016-07-29 20:15:31,762 [Thread-0] INFO balancer.Balancer
(Balancer.java:getLong(231)) - dfs.balancer.movedWinWidth = 2000
(default=5400000)
2016-07-29 20:15:31,763 [Thread-0] INFO balancer.Balancer
(Balancer.java:getInt(249)) - dfs.balancer.moverThreads = 1000 (default=1000)
2016-07-29 20:15:31,763 [Thread-0] INFO balancer.Balancer
(Balancer.java:getInt(249)) - dfs.balancer.dispatcherThreads = 200 (default=200)
2016-07-29 20:15:31,763 [Thread-0] INFO balancer.Balancer
(Balancer.java:getInt(249)) - dfs.datanode.balance.max.concurrent.moves = 50
(default=50)
2016-07-29 20:15:31,763 [Thread-0] INFO balancer.Balancer
(Balancer.java:getLongBytes(240)) - dfs.balancer.getBlocks.size = 2147483648
(default=2147483648)
2016-07-29 20:15:31,763 [Thread-0] INFO balancer.Balancer
(Balancer.java:getLongBytes(240)) - dfs.balancer.getBlocks.min-block-size = 1
(default=10485760)
2016-07-29 20:15:31,763 [Thread-0] INFO balancer.Balancer
(Balancer.java:getLongBytes(240)) - dfs.balancer.max-size-to-move = 10737418240
(default=10737418240)
2016-07-29 20:15:31,763 [Thread-0] INFO balancer.Balancer
(Balancer.java:getLongBytes(240)) - dfs.blocksize = 100 (default=134217728)
2016-07-29 20:15:31,765 [Thread-0] INFO net.NetworkTopology
(NetworkTopology.java:add(435)) - Adding a new node: /rack1/127.0.0.1:65164
2016-07-29 20:15:31,765 [Thread-0] INFO net.NetworkTopology
(NetworkTopology.java:add(435)) - Adding a new node: /rack0/127.0.0.1:65169
2016-07-29 20:15:31,765 [Thread-0] INFO net.NetworkTopology
(NetworkTopology.java:add(435)) - Adding a new node: /rack1/127.0.0.1:65160
2016-07-29 20:15:31,765 [Thread-0] INFO balancer.Balancer
(Balancer.java:logUtilizationCollection(430)) - 1 over-utilized:
[127.0.0.1:65164:DISK]
2016-07-29 20:15:31,765 [Thread-0] INFO balancer.Balancer
(Balancer.java:logUtilizationCollection(430)) - 0 above-average: []
2016-07-29 20:15:31,765 [Thread-0] INFO balancer.Balancer
(Balancer.java:logUtilizationCollection(430)) - 1 below-average:
[127.0.0.1:65160:DISK]
2016-07-29 20:15:31,765 [Thread-0] INFO balancer.Balancer
(Balancer.java:logUtilizationCollection(430)) - 1 underutilized:
[127.0.0.1:65169:DISK]
2016-07-29 20:15:31,765 [Thread-0] INFO balancer.Balancer
(Balancer.java:runOneIteration(603)) - Need to move 533 B to make the cluster
balanced.
2016-07-29 20:15:31,767 [IPC Server handler 3 on 65154] INFO namenode.NameNode
(NameNodeRpcServer.java:rollingUpgrade(1265)) - rollingUpgrade QUERY
2016-07-29 20:15:31,767 [Thread-0] INFO balancer.Balancer
(Balancer.java:chooseStorageGroups(459)) - chooseStorageGroups for SAME_RACK:
overUtilized => underUtilized
2016-07-29 20:15:31,767 [Thread-0] INFO balancer.Balancer
(Balancer.java:chooseStorageGroups(467)) - chooseStorageGroups for SAME_RACK:
overUtilized => belowAvgUtilized
2016-07-29 20:15:31,767 [Thread-0] INFO balancer.Balancer
(Balancer.java:matchSourceWithTargetToMove(525)) - Decided to move 66 B bytes
from 127.0.0.1:65164:DISK to 127.0.0.1:65160:DISK
2016-07-29 20:15:31,767 [Thread-0] INFO balancer.Balancer
(Balancer.java:chooseStorageGroups(475)) - chooseStorageGroups for SAME_RACK:
underUtilized => aboveAvgUtilized
2016-07-29 20:15:31,767 [Thread-0] INFO balancer.Balancer
(Balancer.java:chooseStorageGroups(459)) - chooseStorageGroups for ANY_OTHER:
overUtilized => underUtilized
2016-07-29 20:15:31,768 [Thread-0] INFO balancer.Balancer
(Balancer.java:matchSourceWithTargetToMove(525)) - Decided to move 966 B bytes
from 127.0.0.1:65164:DISK to 127.0.0.1:65169:DISK
2016-07-29 20:15:31,768 [Thread-0] INFO balancer.Balancer
(Balancer.java:chooseStorageGroups(467)) - chooseStorageGroups for ANY_OTHER:
overUtilized => belowAvgUtilized
2016-07-29 20:15:31,768 [Thread-0] INFO balancer.Balancer
(Balancer.java:chooseStorageGroups(475)) - chooseStorageGroups for ANY_OTHER:
underUtilized => aboveAvgUtilized
2016-07-29 20:15:31,768 [Thread-0] INFO balancer.Balancer
(Balancer.java:runOneIteration(627)) - Will move 1.01 KB in this iteration
2016-07-29 20:15:31,771 [pool-44-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741831_1007 with
size=100 from 127.0.0.1:65164:DISK to 127.0.0.1:65160:DISK through
127.0.0.1:65164
2016-07-29 20:15:31,771 [pool-44-thread-2] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741832_1008 with
size=100 from 127.0.0.1:65164:DISK to 127.0.0.1:65160:DISK through
127.0.0.1:65164
2016-07-29 20:15:31,772 [pool-44-thread-3] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741833_1009 with
size=100 from 127.0.0.1:65164:DISK to 127.0.0.1:65160:DISK through
127.0.0.1:65164
2016-07-29 20:15:31,772 [pool-44-thread-4] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741834_1010 with
size=100 from 127.0.0.1:65164:DISK to 127.0.0.1:65160:DISK through
127.0.0.1:65164
2016-07-29 20:15:31,772 [pool-44-thread-5] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741835_1011 with
size=100 from 127.0.0.1:65164:DISK to 127.0.0.1:65160:DISK through
127.0.0.1:65164
2016-07-29 20:15:31,773 [pool-44-thread-6] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741836_1012 with
size=100 from 127.0.0.1:65164:DISK to 127.0.0.1:65160:DISK through
127.0.0.1:65164
2016-07-29 20:15:31,773 [pool-44-thread-7] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741837_1013 with
size=100 from 127.0.0.1:65164:DISK to 127.0.0.1:65160:DISK through
127.0.0.1:65164
2016-07-29 20:15:31,773 [pool-44-thread-8] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741841_1017 with
size=100 from 127.0.0.1:65164:DISK to 127.0.0.1:65160:DISK through
127.0.0.1:65164
2016-07-29 20:15:31,773 [pool-44-thread-9] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741842_1018 with
size=100 from 127.0.0.1:65164:DISK to 127.0.0.1:65160:DISK through
127.0.0.1:65164
2016-07-29 20:15:31,774 [pool-44-thread-10] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741843_1019 with
size=100 from 127.0.0.1:65164:DISK to 127.0.0.1:65160:DISK through
127.0.0.1:65164
2016-07-29 20:15:31,775 [pool-44-thread-11] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(322)) - Start moving blk_1073741844_1020 with
size=100 from 127.0.0.1:65164:DISK to 127.0.0.1:65160:DISK through
127.0.0.1:65164
2016-07-29 20:15:31,775 [DataXceiver for client /127.0.0.1:65333 [Copying block
BP-2103530174-172.17.47.165-1469794498037:blk_1073741831_1007]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-2103530174-172.17.47.165-1469794498037:blk_1073741831_1007 to
/127.0.0.1:65333
2016-07-29 20:15:31,776 [DataXceiver for client /127.0.0.1:65334 [Copying block
BP-2103530174-172.17.47.165-1469794498037:blk_1073741832_1008]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-2103530174-172.17.47.165-1469794498037:blk_1073741832_1008 to
/127.0.0.1:65334
2016-07-29 20:15:31,776 [DataXceiver for client /127.0.0.1:65324 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741832_1008 from
5a87bad2-256e-4dbf-9284-a16f5b75f16c]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-07-29 20:15:31,776 [DataXceiver for client /127.0.0.1:65324 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741832_1008 from
5a87bad2-256e-4dbf-9284-a16f5b75f16c]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-2103530174-172.17.47.165-1469794498037:blk_1073741832_1008 from
/127.0.0.1:65324, delHint=5a87bad2-256e-4dbf-9284-a16f5b75f16c
2016-07-29 20:15:31,776 [DataXceiver for client /127.0.0.1:65323 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741831_1007 from
5a87bad2-256e-4dbf-9284-a16f5b75f16c]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-07-29 20:15:31,777 [DataXceiver for client /127.0.0.1:65323 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741831_1007 from
5a87bad2-256e-4dbf-9284-a16f5b75f16c]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-2103530174-172.17.47.165-1469794498037:blk_1073741831_1007 from
/127.0.0.1:65323, delHint=5a87bad2-256e-4dbf-9284-a16f5b75f16c
2016-07-29 20:15:31,777 [pool-44-thread-2] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741832_1008 with
size=100 from 127.0.0.1:65164:DISK to 127.0.0.1:65160:DISK through
127.0.0.1:65164
2016-07-29 20:15:31,777 [pool-44-thread-1] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741831_1007 with
size=100 from 127.0.0.1:65164:DISK to 127.0.0.1:65160:DISK through
127.0.0.1:65164
2016-07-29 20:15:31,780 [DataXceiver for client /127.0.0.1:65335 [Copying block
BP-2103530174-172.17.47.165-1469794498037:blk_1073741833_1009]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-2103530174-172.17.47.165-1469794498037:blk_1073741833_1009 to
/127.0.0.1:65335
2016-07-29 20:15:31,781 [DataXceiver for client /127.0.0.1:65325 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741833_1009 from
5a87bad2-256e-4dbf-9284-a16f5b75f16c]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-07-29 20:15:31,781 [DataXceiver for client /127.0.0.1:65337 [Copying block
BP-2103530174-172.17.47.165-1469794498037:blk_1073741834_1010]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-2103530174-172.17.47.165-1469794498037:blk_1073741834_1010 to
/127.0.0.1:65337
2016-07-29 20:15:31,781 [DataXceiver for client /127.0.0.1:65326 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741834_1010 from
5a87bad2-256e-4dbf-9284-a16f5b75f16c]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-07-29 20:15:31,781 [DataXceiver for client /127.0.0.1:65325 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741833_1009 from
5a87bad2-256e-4dbf-9284-a16f5b75f16c]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-2103530174-172.17.47.165-1469794498037:blk_1073741833_1009 from
/127.0.0.1:65325, delHint=5a87bad2-256e-4dbf-9284-a16f5b75f16c
2016-07-29 20:15:31,781 [DataXceiver for client /127.0.0.1:65326 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741834_1010 from
5a87bad2-256e-4dbf-9284-a16f5b75f16c]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-2103530174-172.17.47.165-1469794498037:blk_1073741834_1010 from
/127.0.0.1:65326, delHint=5a87bad2-256e-4dbf-9284-a16f5b75f16c
2016-07-29 20:15:31,781 [pool-44-thread-4] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741834_1010 with
size=100 from 127.0.0.1:65164:DISK to 127.0.0.1:65160:DISK through
127.0.0.1:65164
2016-07-29 20:15:31,781 [pool-44-thread-3] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741833_1009 with
size=100 from 127.0.0.1:65164:DISK to 127.0.0.1:65160:DISK through
127.0.0.1:65164
2016-07-29 20:15:31,782 [DataXceiver for client /127.0.0.1:65338 [Copying block
BP-2103530174-172.17.47.165-1469794498037:blk_1073741835_1011]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-2103530174-172.17.47.165-1469794498037:blk_1073741835_1011 to
/127.0.0.1:65338
2016-07-29 20:15:31,782 [DataXceiver for client /127.0.0.1:65327 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741835_1011 from
5a87bad2-256e-4dbf-9284-a16f5b75f16c]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-07-29 20:15:31,782 [DataXceiver for client /127.0.0.1:65339 [Copying block
BP-2103530174-172.17.47.165-1469794498037:blk_1073741836_1012]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-2103530174-172.17.47.165-1469794498037:blk_1073741836_1012 to
/127.0.0.1:65339
2016-07-29 20:15:31,782 [DataXceiver for client /127.0.0.1:65327 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741835_1011 from
5a87bad2-256e-4dbf-9284-a16f5b75f16c]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-2103530174-172.17.47.165-1469794498037:blk_1073741835_1011 from
/127.0.0.1:65327, delHint=5a87bad2-256e-4dbf-9284-a16f5b75f16c
2016-07-29 20:15:31,783 [DataXceiver for client /127.0.0.1:65328 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741836_1012 from
5a87bad2-256e-4dbf-9284-a16f5b75f16c]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-07-29 20:15:31,783 [DataXceiver for client /127.0.0.1:65328 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741836_1012 from
5a87bad2-256e-4dbf-9284-a16f5b75f16c]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-2103530174-172.17.47.165-1469794498037:blk_1073741836_1012 from
/127.0.0.1:65328, delHint=5a87bad2-256e-4dbf-9284-a16f5b75f16c
2016-07-29 20:15:31,783 [DataXceiver for client /127.0.0.1:65340 [Copying block
BP-2103530174-172.17.47.165-1469794498037:blk_1073741837_1013]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-2103530174-172.17.47.165-1469794498037:blk_1073741837_1013 to
/127.0.0.1:65340
2016-07-29 20:15:31,783 [pool-44-thread-5] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741835_1011 with
size=100 from 127.0.0.1:65164:DISK to 127.0.0.1:65160:DISK through
127.0.0.1:65164
2016-07-29 20:15:31,783 [pool-44-thread-6] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741836_1012 with
size=100 from 127.0.0.1:65164:DISK to 127.0.0.1:65160:DISK through
127.0.0.1:65164
2016-07-29 20:15:31,783 [DataXceiver for client /127.0.0.1:65329 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741837_1013 from
5a87bad2-256e-4dbf-9284-a16f5b75f16c]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-07-29 20:15:31,783 [DataXceiver for client /127.0.0.1:65329 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741837_1013 from
5a87bad2-256e-4dbf-9284-a16f5b75f16c]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-2103530174-172.17.47.165-1469794498037:blk_1073741837_1013 from
/127.0.0.1:65329, delHint=5a87bad2-256e-4dbf-9284-a16f5b75f16c
2016-07-29 20:15:31,784 [pool-44-thread-7] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741837_1013 with
size=100 from 127.0.0.1:65164:DISK to 127.0.0.1:65160:DISK through
127.0.0.1:65164
2016-07-29 20:15:31,784 [DataXceiver for client /127.0.0.1:65341 [Copying block
BP-2103530174-172.17.47.165-1469794498037:blk_1073741842_1018]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-2103530174-172.17.47.165-1469794498037:blk_1073741842_1018 to
/127.0.0.1:65341
2016-07-29 20:15:31,784 [DataXceiver for client /127.0.0.1:65331 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741842_1018 from
5a87bad2-256e-4dbf-9284-a16f5b75f16c]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-07-29 20:15:31,784 [DataXceiver for client /127.0.0.1:65331 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741842_1018 from
5a87bad2-256e-4dbf-9284-a16f5b75f16c]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-2103530174-172.17.47.165-1469794498037:blk_1073741842_1018 from
/127.0.0.1:65331, delHint=5a87bad2-256e-4dbf-9284-a16f5b75f16c
2016-07-29 20:15:31,784 [DataXceiver for client /127.0.0.1:65342 [Copying block
BP-2103530174-172.17.47.165-1469794498037:blk_1073741841_1017]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-2103530174-172.17.47.165-1469794498037:blk_1073741841_1017 to
/127.0.0.1:65342
2016-07-29 20:15:31,785 [DataXceiver for client /127.0.0.1:65330 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741841_1017 from
5a87bad2-256e-4dbf-9284-a16f5b75f16c]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-07-29 20:15:31,784 [pool-44-thread-9] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741842_1018 with
size=100 from 127.0.0.1:65164:DISK to 127.0.0.1:65160:DISK through
127.0.0.1:65164
2016-07-29 20:15:31,785 [DataXceiver for client /127.0.0.1:65330 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741841_1017 from
5a87bad2-256e-4dbf-9284-a16f5b75f16c]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-2103530174-172.17.47.165-1469794498037:blk_1073741841_1017 from
/127.0.0.1:65330, delHint=5a87bad2-256e-4dbf-9284-a16f5b75f16c
2016-07-29 20:15:31,785 [pool-44-thread-8] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741841_1017 with
size=100 from 127.0.0.1:65164:DISK to 127.0.0.1:65160:DISK through
127.0.0.1:65164
2016-07-29 20:15:31,787 [DataXceiver for client /127.0.0.1:65343 [Copying block
BP-2103530174-172.17.47.165-1469794498037:blk_1073741843_1019]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-2103530174-172.17.47.165-1469794498037:blk_1073741843_1019 to
/127.0.0.1:65343
2016-07-29 20:15:31,787 [DataXceiver for client /127.0.0.1:65332 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741843_1019 from
5a87bad2-256e-4dbf-9284-a16f5b75f16c]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-07-29 20:15:31,787 [DataXceiver for client /127.0.0.1:65344 [Copying block
BP-2103530174-172.17.47.165-1469794498037:blk_1073741844_1020]] INFO
datanode.DataNode (DataXceiver.java:copyBlock(1054)) - Copied
BP-2103530174-172.17.47.165-1469794498037:blk_1073741844_1020 to
/127.0.0.1:65344
2016-07-29 20:15:31,787 [DataXceiver for client /127.0.0.1:65332 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741843_1019 from
5a87bad2-256e-4dbf-9284-a16f5b75f16c]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-2103530174-172.17.47.165-1469794498037:blk_1073741843_1019 from
/127.0.0.1:65332, delHint=5a87bad2-256e-4dbf-9284-a16f5b75f16c
2016-07-29 20:15:31,787 [DataXceiver for client /127.0.0.1:65336 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741844_1020 from
5a87bad2-256e-4dbf-9284-a16f5b75f16c]] WARN datanode.DataNode
(BlockReceiver.java:<init>(267)) - Could not get file descriptor for
outputstream of class class
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$SimulatedOutputStream
2016-07-29 20:15:31,788 [pool-44-thread-10] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741843_1019 with
size=100 from 127.0.0.1:65164:DISK to 127.0.0.1:65160:DISK through
127.0.0.1:65164
2016-07-29 20:15:31,788 [DataXceiver for client /127.0.0.1:65336 [Replacing
block BP-2103530174-172.17.47.165-1469794498037:blk_1073741844_1020 from
5a87bad2-256e-4dbf-9284-a16f5b75f16c]] INFO datanode.DataNode
(DataXceiver.java:replaceBlock(1175)) - Moved
BP-2103530174-172.17.47.165-1469794498037:blk_1073741844_1020 from
/127.0.0.1:65336, delHint=5a87bad2-256e-4dbf-9284-a16f5b75f16c
2016-07-29 20:15:31,788 [pool-44-thread-11] INFO balancer.Dispatcher
(Dispatcher.java:dispatch(355)) - Successfully moved blk_1073741844_1020 with
size=100 from 127.0.0.1:65164:DISK to 127.0.0.1:65160:DISK through
127.0.0.1:65164
2016-7-29 20:15:32 5 7.32 KB 533 B
1.01 KB
{code}
I looked into the logs, the balancer program always balancing within two nodes
and meanwhile they are in the same rack. It should not reduce the block num
when do the balance operation between two nodes that belong to different rack.
So it seems that was one reason for this.
IMO, one way to optimize this: making the node's rack to the same rack. The
method {{testUnknownDatanode}} is aimed at triggering a situation where one of
the block's location is unknown. So I think we will not influence its logic.
Post a patch for this, I have tested this many times and all of them were
passed whenever there were sleep operation or not. Hi, [~xiaochen], what do you
think?, looking forward to seeing your reply.
> TestBalancer runs timeout intermittently
> ----------------------------------------
>
> Key: HDFS-10602
> URL: https://issues.apache.org/jira/browse/HDFS-10602
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 3.0.0-alpha1
> Reporter: Yiqun Lin
>
> As the jira HDFS-10336 has mentioned, the unit test
> {{TestBalancer#testBalancerWithKeytabs}} will runs too slowly sometimes and
> that leads the timeout. The test {{TestBalancer#testUnknownDatanodeSimple}}
> will also has this problem. These two tests both use the method
> {{testUnknownDatanode}}. We can do some optimization for this method.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
