Auto-Re: [jira] [Commented] (HDFS-8626) Reserved RBW space is not released if creation of RBW File fails
您的邮件已收到！谢谢！
[jira] [Commented] (HDFS-8626) Reserved RBW space is not released if creation of RBW File fails
[
https://issues.apache.org/jira/browse/HDFS-8626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14597957#comment-14597957
]
Arpit Agarwal commented on HDFS-8626:
-
Thanks for the updated test case [~kanaka]. I have committed it to branch-2.7.
Reserved RBW space is not released if creation of RBW File fails
Key: HDFS-8626
URL: https://issues.apache.org/jira/browse/HDFS-8626
Project: Hadoop HDFS
Issue Type: Bug
Reporter: kanaka kumar avvaru
Assignee: kanaka kumar avvaru
Priority: Blocker
Fix For: 2.7.1
Attachments: HDFS-8626-01.patch, HDFS-8626-02.patch,
HDFS-8626-03.patch, HDFS-8626-04.patch, HDFS-8626-04_testcase_on_2.7.patch
The DataNode reserves disk space for a full block when creating an RBW block
and will release the space when the block is finalized (introduced in
HDFS-6898)
But if the RBW file creation fails, the reserved space is not released back.
In a scenario, when the data node disk is full it causes no space left
{{IOException}}. Eventually even if the disk got cleaned, the reserved space
is not release until the Data Node is restarted
Stacktrace for block creation failure is:
{code} IOException in BlockReceiver constructor. Cause is |
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.init(BlockReceiver.java:264)
java.io.IOException: No space left on device
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(File.java:1012)
at
org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.createTmpFile(DatanodeUtil.java:66)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.createRbwFile(BlockPoolSlice.java:274)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbwFile(FsVolumeImpl.java:762)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:1320)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.init(BlockReceiver.java:190)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:678)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:250)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8626) Reserved RBW space is not released if creation of RBW File fails
[
https://issues.apache.org/jira/browse/HDFS-8626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14597264#comment-14597264
]
kanaka kumar avvaru commented on HDFS-8626:
---
Attached patch on 2.7 branch only for the test case. [~arpitagarwal], please
review and commit.
Reserved RBW space is not released if creation of RBW File fails
Key: HDFS-8626
URL: https://issues.apache.org/jira/browse/HDFS-8626
Project: Hadoop HDFS
Issue Type: Bug
Reporter: kanaka kumar avvaru
Assignee: kanaka kumar avvaru
Priority: Blocker
Fix For: 2.7.1
Attachments: HDFS-8626-01.patch, HDFS-8626-02.patch,
HDFS-8626-03.patch, HDFS-8626-04.patch, HDFS-8626-04_testcase_on_2.7.patch
The DataNode reserves disk space for a full block when creating an RBW block
and will release the space when the block is finalized (introduced in
HDFS-6898)
But if the RBW file creation fails, the reserved space is not released back.
In a scenario, when the data node disk is full it causes no space left
{{IOException}}. Eventually even if the disk got cleaned, the reserved space
is not release until the Data Node is restarted
Stacktrace for block creation failure is:
{code} IOException in BlockReceiver constructor. Cause is |
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.init(BlockReceiver.java:264)
java.io.IOException: No space left on device
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(File.java:1012)
at
org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.createTmpFile(DatanodeUtil.java:66)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.createRbwFile(BlockPoolSlice.java:274)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbwFile(FsVolumeImpl.java:762)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:1320)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.init(BlockReceiver.java:190)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:678)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:250)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8626) Reserved RBW space is not released if creation of RBW File fails
[
https://issues.apache.org/jira/browse/HDFS-8626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14595967#comment-14595967
]
kanaka kumar avvaru commented on HDFS-8626:
---
Thanks [~arpitagarwal], for reviews and commit. Right now I don't have compiled
code base for 2.7.. I will upload the test patch on 2.7 some time tomorrow.
Reserved RBW space is not released if creation of RBW File fails
Key: HDFS-8626
URL: https://issues.apache.org/jira/browse/HDFS-8626
Project: Hadoop HDFS
Issue Type: Bug
Reporter: kanaka kumar avvaru
Assignee: kanaka kumar avvaru
Priority: Blocker
Fix For: 2.7.1
Attachments: HDFS-8626-01.patch, HDFS-8626-02.patch,
HDFS-8626-03.patch, HDFS-8626-04.patch
The DataNode reserves disk space for a full block when creating an RBW block
and will release the space when the block is finalized (introduced in
HDFS-6898)
But if the RBW file creation fails, the reserved space is not released back.
In a scenario, when the data node disk is full it causes no space left
{{IOException}}. Eventually even if the disk got cleaned, the reserved space
is not release until the Data Node is restarted
Stacktrace for block creation failure is:
{code} IOException in BlockReceiver constructor. Cause is |
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.init(BlockReceiver.java:264)
java.io.IOException: No space left on device
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(File.java:1012)
at
org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.createTmpFile(DatanodeUtil.java:66)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.createRbwFile(BlockPoolSlice.java:274)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbwFile(FsVolumeImpl.java:762)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:1320)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.init(BlockReceiver.java:190)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:678)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:250)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8626) Reserved RBW space is not released if creation of RBW File fails
[
https://issues.apache.org/jira/browse/HDFS-8626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594678#comment-14594678
]
Hudson commented on HDFS-8626:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk #964 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/964/])
HDFS-8626. Reserved RBW space is not released if creation of RBW File fails.
(Contributed by kanaka kumar avvaru) (arp: rev
55278c012fbd5dcabee0276f8f42d0ab920cec93)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestRbwSpaceReservation.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
Reserved RBW space is not released if creation of RBW File fails
Key: HDFS-8626
URL: https://issues.apache.org/jira/browse/HDFS-8626
Project: Hadoop HDFS
Issue Type: Bug
Reporter: kanaka kumar avvaru
Assignee: kanaka kumar avvaru
Priority: Blocker
Fix For: 2.7.1
Attachments: HDFS-8626-01.patch, HDFS-8626-02.patch,
HDFS-8626-03.patch, HDFS-8626-04.patch
The DataNode reserves disk space for a full block when creating an RBW block
and will release the space when the block is finalized (introduced in
HDFS-6898)
But if the RBW file creation fails, the reserved space is not released back.
In a scenario, when the data node disk is full it causes no space left
{{IOException}}. Eventually even if the disk got cleaned, the reserved space
is not release until the Data Node is restarted
Stacktrace for block creation failure is:
{code} IOException in BlockReceiver constructor. Cause is |
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.init(BlockReceiver.java:264)
java.io.IOException: No space left on device
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(File.java:1012)
at
org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.createTmpFile(DatanodeUtil.java:66)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.createRbwFile(BlockPoolSlice.java:274)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbwFile(FsVolumeImpl.java:762)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:1320)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.init(BlockReceiver.java:190)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:678)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:250)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8626) Reserved RBW space is not released if creation of RBW File fails
[
https://issues.apache.org/jira/browse/HDFS-8626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594686#comment-14594686
]
Hudson commented on HDFS-8626:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #234 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/234/])
HDFS-8626. Reserved RBW space is not released if creation of RBW File fails.
(Contributed by kanaka kumar avvaru) (arp: rev
55278c012fbd5dcabee0276f8f42d0ab920cec93)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestRbwSpaceReservation.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
Reserved RBW space is not released if creation of RBW File fails
Key: HDFS-8626
URL: https://issues.apache.org/jira/browse/HDFS-8626
Project: Hadoop HDFS
Issue Type: Bug
Reporter: kanaka kumar avvaru
Assignee: kanaka kumar avvaru
Priority: Blocker
Fix For: 2.7.1
Attachments: HDFS-8626-01.patch, HDFS-8626-02.patch,
HDFS-8626-03.patch, HDFS-8626-04.patch
The DataNode reserves disk space for a full block when creating an RBW block
and will release the space when the block is finalized (introduced in
HDFS-6898)
But if the RBW file creation fails, the reserved space is not released back.
In a scenario, when the data node disk is full it causes no space left
{{IOException}}. Eventually even if the disk got cleaned, the reserved space
is not release until the Data Node is restarted
Stacktrace for block creation failure is:
{code} IOException in BlockReceiver constructor. Cause is |
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.init(BlockReceiver.java:264)
java.io.IOException: No space left on device
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(File.java:1012)
at
org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.createTmpFile(DatanodeUtil.java:66)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.createRbwFile(BlockPoolSlice.java:274)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbwFile(FsVolumeImpl.java:762)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:1320)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.init(BlockReceiver.java:190)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:678)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:250)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8626) Reserved RBW space is not released if creation of RBW File fails
[
https://issues.apache.org/jira/browse/HDFS-8626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594725#comment-14594725
]
Hudson commented on HDFS-8626:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2162 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2162/])
HDFS-8626. Reserved RBW space is not released if creation of RBW File fails.
(Contributed by kanaka kumar avvaru) (arp: rev
55278c012fbd5dcabee0276f8f42d0ab920cec93)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestRbwSpaceReservation.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
Reserved RBW space is not released if creation of RBW File fails
Key: HDFS-8626
URL: https://issues.apache.org/jira/browse/HDFS-8626
Project: Hadoop HDFS
Issue Type: Bug
Reporter: kanaka kumar avvaru
Assignee: kanaka kumar avvaru
Priority: Blocker
Fix For: 2.7.1
Attachments: HDFS-8626-01.patch, HDFS-8626-02.patch,
HDFS-8626-03.patch, HDFS-8626-04.patch
The DataNode reserves disk space for a full block when creating an RBW block
and will release the space when the block is finalized (introduced in
HDFS-6898)
But if the RBW file creation fails, the reserved space is not released back.
In a scenario, when the data node disk is full it causes no space left
{{IOException}}. Eventually even if the disk got cleaned, the reserved space
is not release until the Data Node is restarted
Stacktrace for block creation failure is:
{code} IOException in BlockReceiver constructor. Cause is |
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.init(BlockReceiver.java:264)
java.io.IOException: No space left on device
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(File.java:1012)
at
org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.createTmpFile(DatanodeUtil.java:66)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.createRbwFile(BlockPoolSlice.java:274)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbwFile(FsVolumeImpl.java:762)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:1320)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.init(BlockReceiver.java:190)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:678)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:250)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8626) Reserved RBW space is not released if creation of RBW File fails
[
https://issues.apache.org/jira/browse/HDFS-8626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594810#comment-14594810
]
Hudson commented on HDFS-8626:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #232 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/232/])
HDFS-8626. Reserved RBW space is not released if creation of RBW File fails.
(Contributed by kanaka kumar avvaru) (arp: rev
55278c012fbd5dcabee0276f8f42d0ab920cec93)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestRbwSpaceReservation.java
Reserved RBW space is not released if creation of RBW File fails
Key: HDFS-8626
URL: https://issues.apache.org/jira/browse/HDFS-8626
Project: Hadoop HDFS
Issue Type: Bug
Reporter: kanaka kumar avvaru
Assignee: kanaka kumar avvaru
Priority: Blocker
Fix For: 2.7.1
Attachments: HDFS-8626-01.patch, HDFS-8626-02.patch,
HDFS-8626-03.patch, HDFS-8626-04.patch
The DataNode reserves disk space for a full block when creating an RBW block
and will release the space when the block is finalized (introduced in
HDFS-6898)
But if the RBW file creation fails, the reserved space is not released back.
In a scenario, when the data node disk is full it causes no space left
{{IOException}}. Eventually even if the disk got cleaned, the reserved space
is not release until the Data Node is restarted
Stacktrace for block creation failure is:
{code} IOException in BlockReceiver constructor. Cause is |
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.init(BlockReceiver.java:264)
java.io.IOException: No space left on device
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(File.java:1012)
at
org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.createTmpFile(DatanodeUtil.java:66)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.createRbwFile(BlockPoolSlice.java:274)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbwFile(FsVolumeImpl.java:762)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:1320)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.init(BlockReceiver.java:190)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:678)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:250)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8626) Reserved RBW space is not released if creation of RBW File fails
[
https://issues.apache.org/jira/browse/HDFS-8626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594743#comment-14594743
]
Hudson commented on HDFS-8626:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #223 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/223/])
HDFS-8626. Reserved RBW space is not released if creation of RBW File fails.
(Contributed by kanaka kumar avvaru) (arp: rev
55278c012fbd5dcabee0276f8f42d0ab920cec93)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestRbwSpaceReservation.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
Reserved RBW space is not released if creation of RBW File fails
Key: HDFS-8626
URL: https://issues.apache.org/jira/browse/HDFS-8626
Project: Hadoop HDFS
Issue Type: Bug
Reporter: kanaka kumar avvaru
Assignee: kanaka kumar avvaru
Priority: Blocker
Fix For: 2.7.1
Attachments: HDFS-8626-01.patch, HDFS-8626-02.patch,
HDFS-8626-03.patch, HDFS-8626-04.patch
The DataNode reserves disk space for a full block when creating an RBW block
and will release the space when the block is finalized (introduced in
HDFS-6898)
But if the RBW file creation fails, the reserved space is not released back.
In a scenario, when the data node disk is full it causes no space left
{{IOException}}. Eventually even if the disk got cleaned, the reserved space
is not release until the Data Node is restarted
Stacktrace for block creation failure is:
{code} IOException in BlockReceiver constructor. Cause is |
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.init(BlockReceiver.java:264)
java.io.IOException: No space left on device
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(File.java:1012)
at
org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.createTmpFile(DatanodeUtil.java:66)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.createRbwFile(BlockPoolSlice.java:274)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbwFile(FsVolumeImpl.java:762)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:1320)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.init(BlockReceiver.java:190)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:678)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:250)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8626) Reserved RBW space is not released if creation of RBW File fails
[
https://issues.apache.org/jira/browse/HDFS-8626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594828#comment-14594828
]
Hudson commented on HDFS-8626:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2180 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2180/])
HDFS-8626. Reserved RBW space is not released if creation of RBW File fails.
(Contributed by kanaka kumar avvaru) (arp: rev
55278c012fbd5dcabee0276f8f42d0ab920cec93)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestRbwSpaceReservation.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
Reserved RBW space is not released if creation of RBW File fails
Key: HDFS-8626
URL: https://issues.apache.org/jira/browse/HDFS-8626
Project: Hadoop HDFS
Issue Type: Bug
Reporter: kanaka kumar avvaru
Assignee: kanaka kumar avvaru
Priority: Blocker
Fix For: 2.7.1
Attachments: HDFS-8626-01.patch, HDFS-8626-02.patch,
HDFS-8626-03.patch, HDFS-8626-04.patch
The DataNode reserves disk space for a full block when creating an RBW block
and will release the space when the block is finalized (introduced in
HDFS-6898)
But if the RBW file creation fails, the reserved space is not released back.
In a scenario, when the data node disk is full it causes no space left
{{IOException}}. Eventually even if the disk got cleaned, the reserved space
is not release until the Data Node is restarted
Stacktrace for block creation failure is:
{code} IOException in BlockReceiver constructor. Cause is |
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.init(BlockReceiver.java:264)
java.io.IOException: No space left on device
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(File.java:1012)
at
org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.createTmpFile(DatanodeUtil.java:66)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.createRbwFile(BlockPoolSlice.java:274)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbwFile(FsVolumeImpl.java:762)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:1320)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.init(BlockReceiver.java:190)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:678)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:250)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8626) Reserved RBW space is not released if creation of RBW File fails
[
https://issues.apache.org/jira/browse/HDFS-8626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594206#comment-14594206
]
Hadoop QA commented on HDFS-8626:
-
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 57s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 33s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 42s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 2m 16s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 32s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 17s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 14s | Pre-build of native portion |
| {color:green}+1{color} | hdfs tests | 162m 16s | Tests passed in hadoop-hdfs.
|
| | | 208m 46s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12740677/HDFS-8626-04.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / b42f1ec |
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11413/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11413/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11413/console |
This message was automatically generated.
Reserved RBW space is not released if creation of RBW File fails
Key: HDFS-8626
URL: https://issues.apache.org/jira/browse/HDFS-8626
Project: Hadoop HDFS
Issue Type: Bug
Reporter: kanaka kumar avvaru
Assignee: kanaka kumar avvaru
Priority: Blocker
Attachments: HDFS-8626-01.patch, HDFS-8626-02.patch,
HDFS-8626-03.patch, HDFS-8626-04.patch
The DataNode reserves disk space for a full block when creating an RBW block
and will release the space when the block is finalized (introduced in
HDFS-6898)
But if the RBW file creation fails, the reserved space is not released back.
In a scenario, when the data node disk is full it causes no space left
{{IOException}}. Eventually even if the disk got cleaned, the reserved space
is not release until the Data Node is restarted
Stacktrace for block creation failure is:
{code} IOException in BlockReceiver constructor. Cause is |
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.init(BlockReceiver.java:264)
java.io.IOException: No space left on device
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(File.java:1012)
at
org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.createTmpFile(DatanodeUtil.java:66)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.createRbwFile(BlockPoolSlice.java:274)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbwFile(FsVolumeImpl.java:762)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:1320)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.init(BlockReceiver.java:190)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:678)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:250)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8626) Reserved RBW space is not released if creation of RBW File fails
[
https://issues.apache.org/jira/browse/HDFS-8626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594280#comment-14594280
]
Hudson commented on HDFS-8626:
--
FAILURE: Integrated in Hadoop-trunk-Commit #8039 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8039/])
HDFS-8626. Reserved RBW space is not released if creation of RBW File fails.
(Contributed by kanaka kumar avvaru) (arp: rev
55278c012fbd5dcabee0276f8f42d0ab920cec93)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestRbwSpaceReservation.java
Reserved RBW space is not released if creation of RBW File fails
Key: HDFS-8626
URL: https://issues.apache.org/jira/browse/HDFS-8626
Project: Hadoop HDFS
Issue Type: Bug
Reporter: kanaka kumar avvaru
Assignee: kanaka kumar avvaru
Priority: Blocker
Fix For: 2.7.1
Attachments: HDFS-8626-01.patch, HDFS-8626-02.patch,
HDFS-8626-03.patch, HDFS-8626-04.patch
The DataNode reserves disk space for a full block when creating an RBW block
and will release the space when the block is finalized (introduced in
HDFS-6898)
But if the RBW file creation fails, the reserved space is not released back.
In a scenario, when the data node disk is full it causes no space left
{{IOException}}. Eventually even if the disk got cleaned, the reserved space
is not release until the Data Node is restarted
Stacktrace for block creation failure is:
{code} IOException in BlockReceiver constructor. Cause is |
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.init(BlockReceiver.java:264)
java.io.IOException: No space left on device
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(File.java:1012)
at
org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.createTmpFile(DatanodeUtil.java:66)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.createRbwFile(BlockPoolSlice.java:274)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbwFile(FsVolumeImpl.java:762)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:1320)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.init(BlockReceiver.java:190)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:678)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:250)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8626) Reserved RBW space is not released if creation of RBW File fails
[
https://issues.apache.org/jira/browse/HDFS-8626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593613#comment-14593613
]
kanaka kumar avvaru commented on HDFS-8626:
---
:) Reflection was working fine [~arpitagarwal] on 0th DN, but file creation was
not failed may be because of multiple DNs in the MiniDFSCluster.
Anyway to avoid confusion, I have updated patch to start single DN and
expecting exception during file creation.
Seems some problem with jenkins due to maintenance shutdown. Will check the
failures if any in the next build.
Reserved RBW space is not released if creation of RBW File fails
Key: HDFS-8626
URL: https://issues.apache.org/jira/browse/HDFS-8626
Project: Hadoop HDFS
Issue Type: Bug
Reporter: kanaka kumar avvaru
Assignee: kanaka kumar avvaru
Priority: Blocker
Attachments: HDFS-8626-01.patch, HDFS-8626-02.patch,
HDFS-8626-03.patch, HDFS-8626-04.patch
The DataNode reserves disk space for a full block when creating an RBW block
and will release the space when the block is finalized (introduced in
HDFS-6898)
But if the RBW file creation fails, the reserved space is not released back.
In a scenario, when the data node disk is full it causes no space left
{{IOException}}. Eventually even if the disk got cleaned, the reserved space
is not release until the Data Node is restarted
Stacktrace for block creation failure is:
{code} IOException in BlockReceiver constructor. Cause is |
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.init(BlockReceiver.java:264)
java.io.IOException: No space left on device
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(File.java:1012)
at
org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.createTmpFile(DatanodeUtil.java:66)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.createRbwFile(BlockPoolSlice.java:274)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbwFile(FsVolumeImpl.java:762)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:1320)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.init(BlockReceiver.java:190)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:678)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:250)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8626) Reserved RBW space is not released if creation of RBW File fails
[
https://issues.apache.org/jira/browse/HDFS-8626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593853#comment-14593853
]
Arpit Agarwal commented on HDFS-8626:
-
Thank you for fixing the test case [~kanaka]! The v4 patch and test look great.
+1 pending Jenkins.
Reserved RBW space is not released if creation of RBW File fails
Key: HDFS-8626
URL: https://issues.apache.org/jira/browse/HDFS-8626
Project: Hadoop HDFS
Issue Type: Bug
Reporter: kanaka kumar avvaru
Assignee: kanaka kumar avvaru
Priority: Blocker
Attachments: HDFS-8626-01.patch, HDFS-8626-02.patch,
HDFS-8626-03.patch, HDFS-8626-04.patch
The DataNode reserves disk space for a full block when creating an RBW block
and will release the space when the block is finalized (introduced in
HDFS-6898)
But if the RBW file creation fails, the reserved space is not released back.
In a scenario, when the data node disk is full it causes no space left
{{IOException}}. Eventually even if the disk got cleaned, the reserved space
is not release until the Data Node is restarted
Stacktrace for block creation failure is:
{code} IOException in BlockReceiver constructor. Cause is |
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.init(BlockReceiver.java:264)
java.io.IOException: No space left on device
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(File.java:1012)
at
org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.createTmpFile(DatanodeUtil.java:66)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.createRbwFile(BlockPoolSlice.java:274)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbwFile(FsVolumeImpl.java:762)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:1320)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.init(BlockReceiver.java:190)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:678)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:250)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8626) Reserved RBW space is not released if creation of RBW File fails
[
https://issues.apache.org/jira/browse/HDFS-8626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14591682#comment-14591682
]
kanaka kumar avvaru commented on HDFS-8626:
---
Right now I don't have unit test case to reproduce but tried stubbing
DatanodeUtil.createTmpFile(...) to throw IOException unconditionally.
Reserved RBW space is not released if creation of RBW File fails
Key: HDFS-8626
URL: https://issues.apache.org/jira/browse/HDFS-8626
Project: Hadoop HDFS
Issue Type: Bug
Reporter: kanaka kumar avvaru
Assignee: kanaka kumar avvaru
Priority: Critical
The DataNode reserves disk space for a full block when creating an RBW block
and will release the space when the block is finalized (introduced in
HDFS-6898)
But if the RBW file creation fails, the reserved space is not released back.
In a scenario, when the data node disk is full it causes no space left
{{IOException}}. Eventually even if the disk got cleaned, the reserved space
is not release until the Data Node is restarted
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8626) Reserved RBW space is not released if creation of RBW File fails
[
https://issues.apache.org/jira/browse/HDFS-8626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14591985#comment-14591985
]
Kihwal Lee commented on HDFS-8626:
--
bq. In a scenario, when the data node disk is full it causes no space left
IOException.
If this was truely the cause, it indicates {{getNextVolume()}} is not working
properly. Normally {{DiskOutOfSpaceException}} is thrown before any space is
reserved. So, if you believe it was caused by {{ENOSPACE}}, try finding out why
the remaining space check was not working. We have seen bad disks causing ext4
to return incorrect values to the du command. In that case, the
{{getAvailable()}} may not return correct result.
In any case, the proposed change is still good for any other types of file
creation failures. [~arpitagarwal], what do you think?
Reserved RBW space is not released if creation of RBW File fails
Key: HDFS-8626
URL: https://issues.apache.org/jira/browse/HDFS-8626
Project: Hadoop HDFS
Issue Type: Bug
Reporter: kanaka kumar avvaru
Assignee: kanaka kumar avvaru
Priority: Critical
Attachments: HDFS-8626-01.patch
The DataNode reserves disk space for a full block when creating an RBW block
and will release the space when the block is finalized (introduced in
HDFS-6898)
But if the RBW file creation fails, the reserved space is not released back.
In a scenario, when the data node disk is full it causes no space left
{{IOException}}. Eventually even if the disk got cleaned, the reserved space
is not release until the Data Node is restarted
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8626) Reserved RBW space is not released if creation of RBW File fails
[
https://issues.apache.org/jira/browse/HDFS-8626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14592917#comment-14592917
]
Hadoop QA commented on HDFS-8626:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 25s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 45s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 57s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 2m 19s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 2m 55s | Post-patch findbugs
hadoop-hdfs-project/hadoop-hdfs compilation is broken. |
| {color:green}+1{color} | findbugs | 2m 55s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 0m 43s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 157m 2s | Tests failed in hadoop-hdfs. |
| | | 201m 39s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.fs.TestHdfsNativeCodeLoader |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12740486/HDFS-8626-03.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 5b5bb8d |
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11409/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11409/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11409/console |
This message was automatically generated.
Reserved RBW space is not released if creation of RBW File fails
Key: HDFS-8626
URL: https://issues.apache.org/jira/browse/HDFS-8626
Project: Hadoop HDFS
Issue Type: Bug
Reporter: kanaka kumar avvaru
Assignee: kanaka kumar avvaru
Priority: Blocker
Attachments: HDFS-8626-01.patch, HDFS-8626-02.patch,
HDFS-8626-03.patch
The DataNode reserves disk space for a full block when creating an RBW block
and will release the space when the block is finalized (introduced in
HDFS-6898)
But if the RBW file creation fails, the reserved space is not released back.
In a scenario, when the data node disk is full it causes no space left
{{IOException}}. Eventually even if the disk got cleaned, the reserved space
is not release until the Data Node is restarted
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8626) Reserved RBW space is not released if creation of RBW File fails
[
https://issues.apache.org/jira/browse/HDFS-8626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14592322#comment-14592322
]
kanaka kumar avvaru commented on HDFS-8626:
---
I am trying to find a way to mock BlockPoolSlice/java.io.File to reproduce it
in junit test case. :( But couldn't do it successfully yet.
Down to File level can't be done as implementation looks different from java 7
8.
If its urgent for 2.7.1, we can go a head,, Otherwise I will try for some time
to add test case.
Reserved RBW space is not released if creation of RBW File fails
Key: HDFS-8626
URL: https://issues.apache.org/jira/browse/HDFS-8626
Project: Hadoop HDFS
Issue Type: Bug
Reporter: kanaka kumar avvaru
Assignee: kanaka kumar avvaru
Priority: Blocker
Attachments: HDFS-8626-01.patch, HDFS-8626-02.patch
The DataNode reserves disk space for a full block when creating an RBW block
and will release the space when the block is finalized (introduced in
HDFS-6898)
But if the RBW file creation fails, the reserved space is not released back.
In a scenario, when the data node disk is full it causes no space left
{{IOException}}. Eventually even if the disk got cleaned, the reserved space
is not release until the Data Node is restarted
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8626) Reserved RBW space is not released if creation of RBW File fails
[
https://issues.apache.org/jira/browse/HDFS-8626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14592153#comment-14592153
]
Arpit Agarwal commented on HDFS-8626:
-
Hi [~kanaka], have you ruled out HDFS-8072? I'll take a look at this patch.
Reserved RBW space is not released if creation of RBW File fails
Key: HDFS-8626
URL: https://issues.apache.org/jira/browse/HDFS-8626
Project: Hadoop HDFS
Issue Type: Bug
Reporter: kanaka kumar avvaru
Assignee: kanaka kumar avvaru
Priority: Blocker
Attachments: HDFS-8626-01.patch
The DataNode reserves disk space for a full block when creating an RBW block
and will release the space when the block is finalized (introduced in
HDFS-6898)
But if the RBW file creation fails, the reserved space is not released back.
In a scenario, when the data node disk is full it causes no space left
{{IOException}}. Eventually even if the disk got cleaned, the reserved space
is not release until the Data Node is restarted
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8626) Reserved RBW space is not released if creation of RBW File fails
[
https://issues.apache.org/jira/browse/HDFS-8626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14592303#comment-14592303
]
Arpit Agarwal commented on HDFS-8626:
-
Thanks for updating the patch.
+1 for the change. Is it straightforward to add a test case?
Reserved RBW space is not released if creation of RBW File fails
Key: HDFS-8626
URL: https://issues.apache.org/jira/browse/HDFS-8626
Project: Hadoop HDFS
Issue Type: Bug
Reporter: kanaka kumar avvaru
Assignee: kanaka kumar avvaru
Priority: Blocker
Attachments: HDFS-8626-01.patch, HDFS-8626-02.patch
The DataNode reserves disk space for a full block when creating an RBW block
and will release the space when the block is finalized (introduced in
HDFS-6898)
But if the RBW file creation fails, the reserved space is not released back.
In a scenario, when the data node disk is full it causes no space left
{{IOException}}. Eventually even if the disk got cleaned, the reserved space
is not release until the Data Node is restarted
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8626) Reserved RBW space is not released if creation of RBW File fails
[
https://issues.apache.org/jira/browse/HDFS-8626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14592149#comment-14592149
]
kanaka kumar avvaru commented on HDFS-8626:
---
Thanks [~kihwal], In that scenario we had only one disk path configured for
data node.
Also for information, get availability on separate Java code test and df
command gives available space from disk as 100+ GB but data node jmx report
shown as 3 GB after few hours also.
In order to confirm the leak , I have ensured through datanode heap dump
analysis that reservedRBW variable in FSVolumeImpl is holding the number
equalent to missing 100+ GB.
Reserved RBW space is not released if creation of RBW File fails
Key: HDFS-8626
URL: https://issues.apache.org/jira/browse/HDFS-8626
Project: Hadoop HDFS
Issue Type: Bug
Reporter: kanaka kumar avvaru
Assignee: kanaka kumar avvaru
Priority: Blocker
Attachments: HDFS-8626-01.patch
The DataNode reserves disk space for a full block when creating an RBW block
and will release the space when the block is finalized (introduced in
HDFS-6898)
But if the RBW file creation fails, the reserved space is not released back.
In a scenario, when the data node disk is full it causes no space left
{{IOException}}. Eventually even if the disk got cleaned, the reserved space
is not release until the Data Node is restarted
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8626) Reserved RBW space is not released if creation of RBW File fails
[
https://issues.apache.org/jira/browse/HDFS-8626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14592209#comment-14592209
]
Brahma Reddy Battula commented on HDFS-8626:
AFAIK, it is different from HDFS-8072..
Reserved RBW space is not released if creation of RBW File fails
Key: HDFS-8626
URL: https://issues.apache.org/jira/browse/HDFS-8626
Project: Hadoop HDFS
Issue Type: Bug
Reporter: kanaka kumar avvaru
Assignee: kanaka kumar avvaru
Priority: Blocker
Attachments: HDFS-8626-01.patch
The DataNode reserves disk space for a full block when creating an RBW block
and will release the space when the block is finalized (introduced in
HDFS-6898)
But if the RBW file creation fails, the reserved space is not released back.
In a scenario, when the data node disk is full it causes no space left
{{IOException}}. Eventually even if the disk got cleaned, the reserved space
is not release until the Data Node is restarted
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8626) Reserved RBW space is not released if creation of RBW File fails
[
https://issues.apache.org/jira/browse/HDFS-8626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14592275#comment-14592275
]
Hadoop QA commented on HDFS-8626:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 48s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 31s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 38s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 2m 11s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 15s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 14s | Pre-build of native portion |
| {color:green}+1{color} | hdfs tests | 162m 30s | Tests passed in hadoop-hdfs.
|
| | | 208m 41s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12740370/HDFS-8626-01.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 2ad6687 |
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11403/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11403/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11403/console |
This message was automatically generated.
Reserved RBW space is not released if creation of RBW File fails
Key: HDFS-8626
URL: https://issues.apache.org/jira/browse/HDFS-8626
Project: Hadoop HDFS
Issue Type: Bug
Reporter: kanaka kumar avvaru
Assignee: kanaka kumar avvaru
Priority: Blocker
Attachments: HDFS-8626-01.patch
The DataNode reserves disk space for a full block when creating an RBW block
and will release the space when the block is finalized (introduced in
HDFS-6898)
But if the RBW file creation fails, the reserved space is not released back.
In a scenario, when the data node disk is full it causes no space left
{{IOException}}. Eventually even if the disk got cleaned, the reserved space
is not release until the Data Node is restarted
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8626) Reserved RBW space is not released if creation of RBW File fails
[
https://issues.apache.org/jira/browse/HDFS-8626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14592177#comment-14592177
]
Arpit Agarwal commented on HDFS-8626:
-
The patch looks good. Nitpick - you can simplify the try block.
{code}
try {
return getBlockPoolSlice(bpid).createRbwFile(b);
} catch (IOException exception) {
{code}
+1 with that fixed.
Reserved RBW space is not released if creation of RBW File fails
Key: HDFS-8626
URL: https://issues.apache.org/jira/browse/HDFS-8626
Project: Hadoop HDFS
Issue Type: Bug
Reporter: kanaka kumar avvaru
Assignee: kanaka kumar avvaru
Priority: Blocker
Attachments: HDFS-8626-01.patch
The DataNode reserves disk space for a full block when creating an RBW block
and will release the space when the block is finalized (introduced in
HDFS-6898)
But if the RBW file creation fails, the reserved space is not released back.
In a scenario, when the data node disk is full it causes no space left
{{IOException}}. Eventually even if the disk got cleaned, the reserved space
is not release until the Data Node is restarted
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8626) Reserved RBW space is not released if creation of RBW File fails
[
https://issues.apache.org/jira/browse/HDFS-8626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14592281#comment-14592281
]
kanaka kumar avvaru commented on HDFS-8626:
---
Thanks for the review [~arpitagarwal], I am aware of HDFS-8072 which is
handling when client terminate while DN receiving the block.
However this issue happens during the construction of block receiver as per the
stack I observed. (Right now I don't have stack.. will update description with
exception stack trace some time tomorrow)
I will wait for the jenkins result before updating the patch for the given
review comment.
Reserved RBW space is not released if creation of RBW File fails
Key: HDFS-8626
URL: https://issues.apache.org/jira/browse/HDFS-8626
Project: Hadoop HDFS
Issue Type: Bug
Reporter: kanaka kumar avvaru
Assignee: kanaka kumar avvaru
Priority: Blocker
Attachments: HDFS-8626-01.patch
The DataNode reserves disk space for a full block when creating an RBW block
and will release the space when the block is finalized (introduced in
HDFS-6898)
But if the RBW file creation fails, the reserved space is not released back.
In a scenario, when the data node disk is full it causes no space left
{{IOException}}. Eventually even if the disk got cleaned, the reserved space
is not release until the Data Node is restarted
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8626) Reserved RBW space is not released if creation of RBW File fails
[
https://issues.apache.org/jira/browse/HDFS-8626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14592300#comment-14592300
]
kanaka kumar avvaru commented on HDFS-8626:
---
:) jenkins gave result while I was writing comment. I have updated the patch
for comment now.
Reserved RBW space is not released if creation of RBW File fails
Key: HDFS-8626
URL: https://issues.apache.org/jira/browse/HDFS-8626
Project: Hadoop HDFS
Issue Type: Bug
Reporter: kanaka kumar avvaru
Assignee: kanaka kumar avvaru
Priority: Blocker
Attachments: HDFS-8626-01.patch, HDFS-8626-02.patch
The DataNode reserves disk space for a full block when creating an RBW block
and will release the space when the block is finalized (introduced in
HDFS-6898)
But if the RBW file creation fails, the reserved space is not released back.
In a scenario, when the data node disk is full it causes no space left
{{IOException}}. Eventually even if the disk got cleaned, the reserved space
is not release until the Data Node is restarted
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8626) Reserved RBW space is not released if creation of RBW File fails
[
https://issues.apache.org/jira/browse/HDFS-8626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14592211#comment-14592211
]
Brahma Reddy Battula commented on HDFS-8626:
[~kanaka] Nice finding...And thanks for working on this..+1 (non-binding )
after addressing [~arpitagarwal] comment.
Reserved RBW space is not released if creation of RBW File fails
Key: HDFS-8626
URL: https://issues.apache.org/jira/browse/HDFS-8626
Project: Hadoop HDFS
Issue Type: Bug
Reporter: kanaka kumar avvaru
Assignee: kanaka kumar avvaru
Priority: Blocker
Attachments: HDFS-8626-01.patch
The DataNode reserves disk space for a full block when creating an RBW block
and will release the space when the block is finalized (introduced in
HDFS-6898)
But if the RBW file creation fails, the reserved space is not released back.
In a scenario, when the data node disk is full it causes no space left
{{IOException}}. Eventually even if the disk got cleaned, the reserved space
is not release until the Data Node is restarted
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8626) Reserved RBW space is not released if creation of RBW File fails
[
https://issues.apache.org/jira/browse/HDFS-8626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14592488#comment-14592488
]
kanaka kumar avvaru commented on HDFS-8626:
---
Updated patch with test case. [~arpitagarwal], please review.
Reserved RBW space is not released if creation of RBW File fails
Key: HDFS-8626
URL: https://issues.apache.org/jira/browse/HDFS-8626
Project: Hadoop HDFS
Issue Type: Bug
Reporter: kanaka kumar avvaru
Assignee: kanaka kumar avvaru
Priority: Blocker
Attachments: HDFS-8626-01.patch, HDFS-8626-02.patch,
HDFS-8626-03.patch
The DataNode reserves disk space for a full block when creating an RBW block
and will release the space when the block is finalized (introduced in
HDFS-6898)
But if the RBW file creation fails, the reserved space is not released back.
In a scenario, when the data node disk is full it causes no space left
{{IOException}}. Eventually even if the disk got cleaned, the reserved space
is not release until the Data Node is restarted
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8626) Reserved RBW space is not released if creation of RBW File fails
[
https://issues.apache.org/jira/browse/HDFS-8626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14592599#comment-14592599
]
Arpit Agarwal commented on HDFS-8626:
-
Hi [~kanaka], thank you for adding the test case. I don't think the reflection
trick in the test is working, else we'd get an exception while writing the file
correct?
Reserved RBW space is not released if creation of RBW File fails
Key: HDFS-8626
URL: https://issues.apache.org/jira/browse/HDFS-8626
Project: Hadoop HDFS
Issue Type: Bug
Reporter: kanaka kumar avvaru
Assignee: kanaka kumar avvaru
Priority: Blocker
Attachments: HDFS-8626-01.patch, HDFS-8626-02.patch,
HDFS-8626-03.patch
The DataNode reserves disk space for a full block when creating an RBW block
and will release the space when the block is finalized (introduced in
HDFS-6898)
But if the RBW file creation fails, the reserved space is not released back.
In a scenario, when the data node disk is full it causes no space left
{{IOException}}. Eventually even if the disk got cleaned, the reserved space
is not release until the Data Node is restarted
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
