[
https://issues.apache.org/jira/browse/HDFS-6489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214049#comment-15214049
]
Weiwei Yang commented on HDFS-6489:
-----------------------------------
Most of test failures are timeout, I can run them successfully on my laptop,
the only failure was
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica
{quote}
Running org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.861 sec - in
org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser
Running org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.426 sec - in
org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer
Running org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica
Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 16.789 sec <<<
FAILURE! - in
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica
testAppend(org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica)
Time elapsed: 2.146 sec <<< FAILURE!
java.lang.AssertionError: Should not have space to append to an RWR
replicaBP-228064733-9.181.90.49-1459147940642:blk_4_2004
at org.junit.Assert.fail(Assert.java:88)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica.testAppend(TestWriteToReplica.java:181)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica.testAppend(TestWriteToReplica.java:95)
Running org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 255.421 sec -
in org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner
Running org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 69.761 sec - in
org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
Running org.apache.hadoop.hdfs.server.datanode.TestDataNodeUUID
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.381 sec - in
org.apache.hadoop.hdfs.server.datanode.TestDataNodeUUID
Running org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 36.823 sec - in
org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
Tests run: 35, Failures: 1, Errors: 0, Skipped: 0
{quote}
this wasn't a code issue, this test was based on the old logic, I will upload
v3 patch shortly to resolve this.
> DFS Used space is not correct computed on frequent append operations
> --------------------------------------------------------------------
>
> Key: HDFS-6489
> URL: https://issues.apache.org/jira/browse/HDFS-6489
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
> Affects Versions: 2.2.0, 2.7.1, 2.7.2
> Reporter: stanley shi
> Assignee: Weiwei Yang
> Attachments: HDFS-6489.001.patch, HDFS-6489.002.patch, HDFS6489.java
>
>
> The current implementation of the Datanode will increase the DFS used space
> on each block write operation. This is correct in most scenario (create new
> file), but sometimes it will behave in-correct(append small data to a large
> block).
> For example, I have a file with only one block(say, 60M). Then I try to
> append to it very frequently but each time I append only 10 bytes;
> Then on each append, dfs used will be increased with the length of the
> block(60M), not teh actual data length(10bytes).
> Consider in a scenario I use many clients to append concurrently to a large
> number of files (1000+), assume the block size is 32M (half of the default
> value), then the dfs used will be increased 1000*32M = 32G on each append to
> the files; but actually I only write 10K bytes; this will cause the datanode
> to report in-sufficient disk space on data write.
> {quote}2014-06-04 15:27:34,719 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: opWriteBlock
> BP-1649188734-10.37.7.142-1398844098971:blk_1073742834_45306 received
> exception org.apache.hadoop.util.DiskChecker$DiskOutOfSpaceException:
> Insufficient space for appending to FinalizedReplica, blk_1073742834_45306,
> FINALIZED{quote}
> But the actual disk usage:
> {quote}
> [root@hdsh143 ~]# df -h
> Filesystem Size Used Avail Use% Mounted on
> /dev/sda3 16G 2.9G 13G 20% /
> tmpfs 1.9G 72K 1.9G 1% /dev/shm
> /dev/sda1 97M 32M 61M 35% /boot
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
