[
https://issues.apache.org/jira/browse/HADOOP-3707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12612670#action_12612670
]
Konstantin Shvachko commented on HADOOP-3707:
---------------------------------------------
This is a nice idea, but I am not happy that
- it is rather complex,
- it is very approximate,
- it is not localized in one place of the code, and therefore hard to maintain.
One disadvantage of the approach is that the name-node forgets about scheduled
replication after a while (10 min?).
Which means that if you do a lot of slow writes or replications then you can
schedule a lot for one node then forget about itand then schedule the same
amount again.
# I think you don't need the approximation , and hence can remove
prevApproxBlockScheduled, because we know exactly when to increment and when to
decrement the scheduling counters.
#- increment when block is scheduled for replication and when a new block is
allocated, this can be in one common place at chooseTargets().
#- decrement when blockReceived() and when the block moves from
pendingReplications back to neededReplications. Need to find one common place
for the call.
# May be we can solve this using simpler things. Like
#- slow down the replication scheduler.
#- use bigger values for dfs.datanode.du.pct or dfs.datanode.du.reserved.
> Frequent DiskOutOfSpaceException on almost-full datanodes
> ---------------------------------------------------------
>
> Key: HADOOP-3707
> URL: https://issues.apache.org/jira/browse/HADOOP-3707
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.0
> Reporter: Koji Noguchi
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.17.2, 0.18.0, 0.19.0
>
> Attachments: HADOOP-3707-branch-017.patch,
> HADOOP-3707-branch-017.patch, HADOOP-3707-trunk.patch,
> HADOOP-3707-trunk.patch, HADOOP-3707-trunk.patch
>
>
> On a datanode which is completely full (leaving reserve space), we
> frequently see
> target node reporting,
> {noformat}
> 2008-07-07 16:54:44,707 INFO org.apache.hadoop.dfs.DataNode: Receiving block
> blk_3328886742742952100 src: /11.1.11.111:22222 dest: /11.1.11.111:22222
> 2008-07-07 16:54:44,708 INFO org.apache.hadoop.dfs.DataNode: writeBlock
> blk_3328886742742952100 received exception
> org.apache.hadoop.util.DiskChecker$DiskOutOfSpaceException: Insufficient
> space for an additional block
> 2008-07-07 16:54:44,708 ERROR org.apache.hadoop.dfs.DataNode:
> 33.3.33.33:22222:DataXceiver:
> org.apache.hadoop.util.DiskChecker$DiskOutOfSpaceException: Insufficient
> space for an additional block
> at
> org.apache.hadoop.dfs.FSDataset$FSVolumeSet.getNextVolume(FSDataset.java:444)
> at org.apache.hadoop.dfs.FSDataset.writeToBlock(FSDataset.java:716)
> at
> org.apache.hadoop.dfs.DataNode$BlockReceiver.<init>(DataNode.java:2187)
> at
> org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:1113)
> at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:976)
> at java.lang.Thread.run(Thread.java:619)
> {noformat}
> Sender reporting
> {noformat}
> 2008-07-07 16:54:44,712 INFO org.apache.hadoop.dfs.DataNode:
> 11.1.11.111:22222:Exception writing block blk_3328886742742952100 to mirror
> 33.3.33.33:22222
> java.io.IOException: Broken pipe
> at sun.nio.ch.FileDispatcher.write0(Native Method)
> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)
> at sun.nio.ch.IOUtil.write(IOUtil.java:75)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
> at
> org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:53)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:140)
> at
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:144)
> at
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:105)
> at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at
> org.apache.hadoop.dfs.DataNode$BlockReceiver.receiveChunk(DataNode.java:2292)
> at
> org.apache.hadoop.dfs.DataNode$BlockReceiver.receivePacket(DataNode.java:2411)
> at
> org.apache.hadoop.dfs.DataNode$BlockReceiver.receiveBlock(DataNode.java:2476)
> at
> org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:1204)
> at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:976)
> at java.lang.Thread.run(Thread.java:619)
> {noformat}
> Since it's not constantly happening, my guess is whenever datanode gets some
> small space available, namenode over-assigns blocks which can fail the block
> pipeline.
> (Note, before 0.17, namenode was much slower in assigning blocks)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
