[
https://issues.apache.org/jira/browse/HDFS-3510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13290603#comment-13290603
]
Colin Patrick McCabe commented on HDFS-3510:
--------------------------------------------
bq. writeFully: should probably take a WritableByteChannel, not a FileChannel
I did consider that, but unfortunately the method I need is
FileChannel(ByteBuffer, long), which isn't present in the superclass. (The
superclass only has WritableByteChannel(ByteBuffer), which updates the current
offset, something we need to avoid doing in this code.)
bq. tiledFill: I dont think this is general-purpose enough to merit inclusion
in IOUtils.
Yeah, I guess... it seemed so clear to me, but then writing the JavaDoc I
started to have doubts about how generic it really was :)
bq. Seems like some of these changes are not present in trunk, so we should fix
them in trunk and then backport.
Yeah, that makes sense. I'll create a JIRA for implementing writeFully and do
that in trunk. It's general-purpose enough to be worth its own JIRA.
I think trunk might have the "We only try to do preallocation when the current
position is less than 4096 bytes from the end of the file" problem, so maybe
that is also worth putting into a separate JIRA.
Hopefully I can get those done and backported fast...
> FSEditLog pre-allocation does not work in branch-1
> --------------------------------------------------
>
> Key: HDFS-3510
> URL: https://issues.apache.org/jira/browse/HDFS-3510
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Fix For: 1.0.0
>
> Attachments: HDFS-3510-b1.001.patch, HDFS-3510-b1.002.patch
>
>
> In the FSEditLog, we want to avoid running out of space in the middle of
> writing an edit log operation to the disk. We do this by a process called
> "preallocation"-- reserving space on the disk for the upcoming edit log
> entries before beginning to write them.
> branch-1 has some major problems with the way it does preallocation. These
> problems can lead to corrupt edit logs when the disk runs out of space during
> an edit log sync operation.
> The problems are:
> * We use FileChannel#write without checking for short writes, but
> WritableByteChannel explicitly documents that they are possible, and the
> FileChannel subclass is silent on the issue.
> * We only try to do preallocation when the current position is less than 4096
> bytes from the end of the file. However, bufReady starts out at 512kb, and
> only gets bigger from there. There is no way that 4kb is enough space to
> reserve.
> * The current code seems to be based on a misunderstanding of how space is
> allocated in files in Linux. In FileChannel#write(ByteBuffer, long), the
> second argument is the offset to start writing at. Since we set this to
> fc.position() + 1024*1024, this means that we *start* writing a megabyte
> after the end of the file. This is guaranteed to create a sparse file on
> Linux, defeating the point of pre-allocation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
