[
https://issues.apache.org/jira/browse/HDFS-1846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13024706#comment-13024706
]
Brock Noland commented on HDFS-1846:
------------------------------------
This is a major improvement to the existing preallocation scheme. If we decide
the time to preallocate is too costly, then I think I the existing scheme
should be scrapped. Sparse files are not preallocated. You can create a file
which appears to be 500GB instantly by seeking past the end of the file.
# rm -f test; time dd if=/dev/null of=test seek=999999999; ls -lh test
0+0 records in
0+0 records out
0 bytes (0 B) copied, 1.4e-05 seconds, 0.0 kB/s
real 0m0.002s
user 0m0.000s
sys 0m0.002s
-rw-r--r-- 1 root root 477G Apr 24 21:04 test
> Don't fill preallocated portion of edits log with 0x00
> ------------------------------------------------------
>
> Key: HDFS-1846
> URL: https://issues.apache.org/jira/browse/HDFS-1846
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Aaron T. Myers
> Assignee: Aaron T. Myers
> Attachments: hdfs-1846-perf-analysis.0.patch, hdfs-1846.0.txt
>
>
> HADOOP-2330 added a feature to preallocate space in the local file system for
> the NN transaction log. That change seeks past the current end of the file
> and writes out some data, which on most systems results in the intervening
> data in the file being filled with zeros. Most underlying file systems have
> special handling for sparse files, and don't actually allocate blocks on disk
> for blocks of a file which consist completely of 0x00.
> I've seen cases in the wild where the volume an edits dir is on fills up,
> resulting in a partial final transaction being written out to disk. If you
> examine the bytes of this (now corrupt) edits file, you'll see the partial
> final transaction followed by a lot of zeros, suggesting that the
> preallocation previously succeeded before the volume ran out of space. If we
> fill the preallocated space with something other than zeros, we'd likely see
> the failure at preallocation time, rather than transaction-writing time, and
> so cause the NN to crash earlier, without a partial transaction being written
> out.
> I also hypothesize that filling the preallocated space in the edits log with
> something other than 0x00 will result in a performance improvement in NN
> throughput. I haven't tested this yet, but I intend to as part of this JIRA.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
