[jira] [Commented] (HDFS-6898) DN must reserve space for a full block when an RBW block is created
[ https://issues.apache.org/jira/browse/HDFS-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14592259#comment-14592259 ] Arpit Agarwal commented on HDFS-6898: - 2.6.0 is correct, sorry about the oversight. DN must reserve space for a full block when an RBW block is created --- Key: HDFS-6898 URL: https://issues.apache.org/jira/browse/HDFS-6898 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 2.5.0 Reporter: Gopal V Assignee: Arpit Agarwal Fix For: 2.6.0 Attachments: HDFS-6898.01.patch, HDFS-6898.03.patch, HDFS-6898.04.patch, HDFS-6898.05.patch, HDFS-6898.06.patch, HDFS-6898.07.patch DN will successfully create two RBW blocks on the same volume even if the free space is sufficient for just one full block. One or both block writers may subsequently get a DiskOutOfSpace exception. This can be avoided by allocating space up front. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6898) DN must reserve space for a full block when an RBW block is created
[ https://issues.apache.org/jira/browse/HDFS-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124868#comment-14124868 ] Hudson commented on HDFS-6898: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #673 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/673/]) HDFS-6898. DN must reserve space for a full block when an RBW block is created. (Contributed by Arpit Agarwal) (arp: rev d1fa58292e87bc29b4ef1278368c2be938a0afc4) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInPipeline.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaBeingWritten.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestRbwSpaceReservation.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestWriteToReplica.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java DN must reserve space for a full block when an RBW block is created --- Key: HDFS-6898 URL: https://issues.apache.org/jira/browse/HDFS-6898 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 2.5.0 Reporter: Gopal V Assignee: Arpit Agarwal Attachments: HDFS-6898.01.patch, HDFS-6898.03.patch, HDFS-6898.04.patch, HDFS-6898.05.patch, HDFS-6898.06.patch, HDFS-6898.07.patch DN will successfully create two RBW blocks on the same volume even if the free space is sufficient for just one full block. One or both block writers may subsequently get a DiskOutOfSpace exception. This can be avoided by allocating space up front. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6898) DN must reserve space for a full block when an RBW block is created
[ https://issues.apache.org/jira/browse/HDFS-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124900#comment-14124900 ] Hudson commented on HDFS-6898: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1864 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1864/]) HDFS-6898. DN must reserve space for a full block when an RBW block is created. (Contributed by Arpit Agarwal) (arp: rev d1fa58292e87bc29b4ef1278368c2be938a0afc4) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestRbwSpaceReservation.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestWriteToReplica.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaBeingWritten.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInPipeline.java DN must reserve space for a full block when an RBW block is created --- Key: HDFS-6898 URL: https://issues.apache.org/jira/browse/HDFS-6898 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 2.5.0 Reporter: Gopal V Assignee: Arpit Agarwal Attachments: HDFS-6898.01.patch, HDFS-6898.03.patch, HDFS-6898.04.patch, HDFS-6898.05.patch, HDFS-6898.06.patch, HDFS-6898.07.patch DN will successfully create two RBW blocks on the same volume even if the free space is sufficient for just one full block. One or both block writers may subsequently get a DiskOutOfSpace exception. This can be avoided by allocating space up front. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6898) DN must reserve space for a full block when an RBW block is created
[ https://issues.apache.org/jira/browse/HDFS-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124908#comment-14124908 ] Hudson commented on HDFS-6898: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1889 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1889/]) HDFS-6898. DN must reserve space for a full block when an RBW block is created. (Contributed by Arpit Agarwal) (arp: rev d1fa58292e87bc29b4ef1278368c2be938a0afc4) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestWriteToReplica.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestRbwSpaceReservation.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaBeingWritten.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInPipeline.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java DN must reserve space for a full block when an RBW block is created --- Key: HDFS-6898 URL: https://issues.apache.org/jira/browse/HDFS-6898 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 2.5.0 Reporter: Gopal V Assignee: Arpit Agarwal Attachments: HDFS-6898.01.patch, HDFS-6898.03.patch, HDFS-6898.04.patch, HDFS-6898.05.patch, HDFS-6898.06.patch, HDFS-6898.07.patch DN will successfully create two RBW blocks on the same volume even if the free space is sufficient for just one full block. One or both block writers may subsequently get a DiskOutOfSpace exception. This can be avoided by allocating space up front. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6898) DN must reserve space for a full block when an RBW block is created
[
https://issues.apache.org/jira/browse/HDFS-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124694#comment-14124694
]
Hadoop QA commented on HDFS-6898:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12667038/HDFS-6898.07.patch
against trunk revision 88209ce.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7930//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7930//console
This message is automatically generated.
DN must reserve space for a full block when an RBW block is created
---
Key: HDFS-6898
URL: https://issues.apache.org/jira/browse/HDFS-6898
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.5.0
Reporter: Gopal V
Assignee: Arpit Agarwal
Attachments: HDFS-6898.01.patch, HDFS-6898.03.patch,
HDFS-6898.04.patch, HDFS-6898.05.patch, HDFS-6898.06.patch, HDFS-6898.07.patch
DN will successfully create two RBW blocks on the same volume even if the
free space is sufficient for just one full block.
One or both block writers may subsequently get a DiskOutOfSpace exception.
This can be avoided by allocating space up front.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6898) DN must reserve space for a full block when an RBW block is created
[
https://issues.apache.org/jira/browse/HDFS-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124746#comment-14124746
]
Chris Nauroth commented on HDFS-6898:
-
+1 for patch v7 too. The failure in {{TestPipelinesFailover}} is unrelated and
tracked elsewhere. Thanks again, Arpit.
DN must reserve space for a full block when an RBW block is created
---
Key: HDFS-6898
URL: https://issues.apache.org/jira/browse/HDFS-6898
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.5.0
Reporter: Gopal V
Assignee: Arpit Agarwal
Attachments: HDFS-6898.01.patch, HDFS-6898.03.patch,
HDFS-6898.04.patch, HDFS-6898.05.patch, HDFS-6898.06.patch, HDFS-6898.07.patch
DN will successfully create two RBW blocks on the same volume even if the
free space is sufficient for just one full block.
One or both block writers may subsequently get a DiskOutOfSpace exception.
This can be avoided by allocating space up front.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6898) DN must reserve space for a full block when an RBW block is created
[
https://issues.apache.org/jira/browse/HDFS-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123676#comment-14123676
]
Chris Nauroth commented on HDFS-6898:
-
Hi, [~arpitagarwal]. The patch looks great. I have just one comment. In
{{FsVolumeImpl#releaseReservedSpace}}, the failsafe logic could be subject to a
data race. If the {{addAndGet}} results in a negative value, and then another
thread calls {{reserveSpaceForRbw}} before the reset to 0 executes, then we'd
lose that second thread's reservation. Another approach might be to use a loop
that calculates the new value (or 0) and makes a single call to
{{compareAndSet}}, repeating until successful.
DN must reserve space for a full block when an RBW block is created
---
Key: HDFS-6898
URL: https://issues.apache.org/jira/browse/HDFS-6898
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.5.0
Reporter: Gopal V
Assignee: Arpit Agarwal
Attachments: HDFS-6898.01.patch, HDFS-6898.03.patch,
HDFS-6898.04.patch, HDFS-6898.05.patch
DN will successfully create two RBW blocks on the same volume even if the
free space is sufficient for just one full block.
One or both block writers may subsequently get a DiskOutOfSpace exception.
This can be avoided by allocating space up front.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6898) DN must reserve space for a full block when an RBW block is created
[
https://issues.apache.org/jira/browse/HDFS-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124267#comment-14124267
]
Hadoop QA commented on HDFS-6898:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12666935/HDFS-6898.06.patch
against trunk revision 21c0cde.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.datanode.TestMultipleNNDataBlockScanner
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestRbwSpaceReservation
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7924//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7924//console
This message is automatically generated.
DN must reserve space for a full block when an RBW block is created
---
Key: HDFS-6898
URL: https://issues.apache.org/jira/browse/HDFS-6898
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.5.0
Reporter: Gopal V
Assignee: Arpit Agarwal
Attachments: HDFS-6898.01.patch, HDFS-6898.03.patch,
HDFS-6898.04.patch, HDFS-6898.05.patch, HDFS-6898.06.patch
DN will successfully create two RBW blocks on the same volume even if the
free space is sufficient for just one full block.
One or both block writers may subsequently get a DiskOutOfSpace exception.
This can be avoided by allocating space up front.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6898) DN must reserve space for a full block when an RBW block is created
[ https://issues.apache.org/jira/browse/HDFS-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117730#comment-14117730 ] Arpit Agarwal commented on HDFS-6898: - [~cmccabe], did you get a chance to take a look at the updated patch? DN must reserve space for a full block when an RBW block is created --- Key: HDFS-6898 URL: https://issues.apache.org/jira/browse/HDFS-6898 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 2.5.0 Reporter: Gopal V Assignee: Arpit Agarwal Attachments: HDFS-6898.01.patch, HDFS-6898.03.patch, HDFS-6898.04.patch, HDFS-6898.05.patch DN will successfully create two RBW blocks on the same volume even if the free space is sufficient for just one full block. One or both block writers may subsequently get a DiskOutOfSpace exception. This can be avoided by allocating space up front. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6898) DN must reserve space for a full block when an RBW block is created
[ https://issues.apache.org/jira/browse/HDFS-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14115486#comment-14115486 ] Arpit Agarwal commented on HDFS-6898: - The zero is being passed in convertTemporaryToRbw, not in append. We pass zero for temporary blocks since we don't know the estimated max block length. DN must reserve space for a full block when an RBW block is created --- Key: HDFS-6898 URL: https://issues.apache.org/jira/browse/HDFS-6898 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 2.5.0 Reporter: Gopal V Assignee: Arpit Agarwal Attachments: HDFS-6898.01.patch, HDFS-6898.03.patch, HDFS-6898.04.patch, HDFS-6898.05.patch DN will successfully create two RBW blocks on the same volume even if the free space is sufficient for just one full block. One or both block writers may subsequently get a DiskOutOfSpace exception. This can be avoided by allocating space up front. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6898) DN must reserve space for a full block when an RBW block is created
[ https://issues.apache.org/jira/browse/HDFS-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114824#comment-14114824 ] Zesheng Wu commented on HDFS-6898: -- [~arpitagarwal], what about append data to existing blocks, should we also need to reserve space? DN must reserve space for a full block when an RBW block is created --- Key: HDFS-6898 URL: https://issues.apache.org/jira/browse/HDFS-6898 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 2.5.0 Reporter: Gopal V Assignee: Arpit Agarwal Attachments: HDFS-6898.01.patch, HDFS-6898.03.patch, HDFS-6898.04.patch, HDFS-6898.05.patch DN will successfully create two RBW blocks on the same volume even if the free space is sufficient for just one full block. One or both block writers may subsequently get a DiskOutOfSpace exception. This can be avoided by allocating space up front. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6898) DN must reserve space for a full block when an RBW block is created
[ https://issues.apache.org/jira/browse/HDFS-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114827#comment-14114827 ] Arpit Agarwal commented on HDFS-6898: - Yes the patch reserves space for appends. We reserve (maxBlockLength - currentBlockLength) bytes when a block is opened for append. DN must reserve space for a full block when an RBW block is created --- Key: HDFS-6898 URL: https://issues.apache.org/jira/browse/HDFS-6898 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 2.5.0 Reporter: Gopal V Assignee: Arpit Agarwal Attachments: HDFS-6898.01.patch, HDFS-6898.03.patch, HDFS-6898.04.patch, HDFS-6898.05.patch DN will successfully create two RBW blocks on the same volume even if the free space is sufficient for just one full block. One or both block writers may subsequently get a DiskOutOfSpace exception. This can be avoided by allocating space up front. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6898) DN must reserve space for a full block when an RBW block is created
[
https://issues.apache.org/jira/browse/HDFS-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114847#comment-14114847
]
Zesheng Wu commented on HDFS-6898:
--
Oh, sorry, I missed that detail in the patch.
One more question, in the append() method :
{code}
ReplicaBeingWritten newReplicaInfo = new ReplicaBeingWritten(
replicaInfo.getBlockId(), replicaInfo.getNumBytes(), newGS,
v, newBlkFile.getParentFile(), Thread.currentThread(),
estimateBlockLen);
{code}
Here reserves {{estimateBlockLen}} bytes
{code}
v.reserveSpaceForRbw(estimateBlockLen - replicaInfo.getNumBytes());
{code}
Here reserves {{estimateBlockLen - replicaInfo.getNumBytes()}} bytes
What's the difference between these two?
DN must reserve space for a full block when an RBW block is created
---
Key: HDFS-6898
URL: https://issues.apache.org/jira/browse/HDFS-6898
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.5.0
Reporter: Gopal V
Assignee: Arpit Agarwal
Attachments: HDFS-6898.01.patch, HDFS-6898.03.patch,
HDFS-6898.04.patch, HDFS-6898.05.patch
DN will successfully create two RBW blocks on the same volume even if the
free space is sufficient for just one full block.
One or both block writers may subsequently get a DiskOutOfSpace exception.
This can be avoided by allocating space up front.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6898) DN must reserve space for a full block when an RBW block is created
[
https://issues.apache.org/jira/browse/HDFS-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14110554#comment-14110554
]
Hadoop QA commented on HDFS-6898:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12664334/HDFS-6898.04.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestRbwSpaceReservation
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7765//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7765//console
This message is automatically generated.
DN must reserve space for a full block when an RBW block is created
---
Key: HDFS-6898
URL: https://issues.apache.org/jira/browse/HDFS-6898
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.5.0
Reporter: Gopal V
Assignee: Arpit Agarwal
Attachments: HDFS-6898.01.patch, HDFS-6898.03.patch,
HDFS-6898.04.patch, HDFS-6898.05.patch
DN will successfully create two RBW blocks on the same volume even if the
free space is sufficient for just one full block.
One or both block writers may subsequently get a DiskOutOfSpace exception.
This can be avoided by allocating space up front.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6898) DN must reserve space for a full block when an RBW block is created
[
https://issues.apache.org/jira/browse/HDFS-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14110563#comment-14110563
]
Hadoop QA commented on HDFS-6898:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12664335/HDFS-6898.05.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestPersistBlocks
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7766//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7766//console
This message is automatically generated.
DN must reserve space for a full block when an RBW block is created
---
Key: HDFS-6898
URL: https://issues.apache.org/jira/browse/HDFS-6898
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.5.0
Reporter: Gopal V
Assignee: Arpit Agarwal
Attachments: HDFS-6898.01.patch, HDFS-6898.03.patch,
HDFS-6898.04.patch, HDFS-6898.05.patch
DN will successfully create two RBW blocks on the same volume even if the
free space is sufficient for just one full block.
One or both block writers may subsequently get a DiskOutOfSpace exception.
This can be avoided by allocating space up front.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6898) DN must reserve space for a full block when an RBW block is created
[ https://issues.apache.org/jira/browse/HDFS-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14110632#comment-14110632 ] Vinayakumar B commented on HDFS-6898: - Do you think this reservation should be done for the tmp files also? DN must reserve space for a full block when an RBW block is created --- Key: HDFS-6898 URL: https://issues.apache.org/jira/browse/HDFS-6898 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 2.5.0 Reporter: Gopal V Assignee: Arpit Agarwal Attachments: HDFS-6898.01.patch, HDFS-6898.03.patch, HDFS-6898.04.patch, HDFS-6898.05.patch DN will successfully create two RBW blocks on the same volume even if the free space is sufficient for just one full block. One or both block writers may subsequently get a DiskOutOfSpace exception. This can be avoided by allocating space up front. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6898) DN must reserve space for a full block when an RBW block is created
[ https://issues.apache.org/jira/browse/HDFS-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14111072#comment-14111072 ] Arpit Agarwal commented on HDFS-6898: - Yes it may be helpful to have reservation for tmp files also. I'll file a separate Jira to look into it. DN must reserve space for a full block when an RBW block is created --- Key: HDFS-6898 URL: https://issues.apache.org/jira/browse/HDFS-6898 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 2.5.0 Reporter: Gopal V Assignee: Arpit Agarwal Attachments: HDFS-6898.01.patch, HDFS-6898.03.patch, HDFS-6898.04.patch, HDFS-6898.05.patch DN will successfully create two RBW blocks on the same volume even if the free space is sufficient for just one full block. One or both block writers may subsequently get a DiskOutOfSpace exception. This can be avoided by allocating space up front. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6898) DN must reserve space for a full block when an RBW block is created
[
https://issues.apache.org/jira/browse/HDFS-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14109898#comment-14109898
]
Colin Patrick McCabe commented on HDFS-6898:
Nice idea. I will wait for the revised version of this patch.
bq. A related question is whether DN should pre-allocate disk space for
received RBW blocks. If the underlying local FS supports fallocate, this will
improve the disk layout of the block.
We certainly don't want to call {{fallocate}}, since that will result in
hundreds of megs of unnecessary disk I/O. We could call {{ftruncate}}, but
this would get confusing if we didn't end up writing the full block. Currently
I believe there are some places assuming that the block size means
something.. i.e. that we don't just set all block sizes to 128 MB and use
sparse files. So it would be pretty complex to reverse that assumption.
I doubt there would be much performance gain since we're already doing big
sequential writes, a good case for ext4, etc.
DN must reserve space for a full block when an RBW block is created
---
Key: HDFS-6898
URL: https://issues.apache.org/jira/browse/HDFS-6898
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.5.0
Reporter: Gopal V
Assignee: Arpit Agarwal
Attachments: HDFS-6898.01.patch, HDFS-6898.03.patch
DN will successfully create two RBW blocks on the same volume even if the
free space is sufficient for just one full block.
One or both block writers may subsequently get a DiskOutOfSpace exception.
This can be avoided by allocating space up front.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6898) DN must reserve space for a full block when an RBW block is created
[
https://issues.apache.org/jira/browse/HDFS-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14109909#comment-14109909
]
Zhe Zhang commented on HDFS-6898:
-
[~cmccabe] Will {{fallocate}} trigger disk I/O? I thought it only manipulates
metadata: http://linux.die.net/man/1/fallocate -- this is done quickly by
allocating blocks and marking them as uninitialized, requiring no IO to the
data blocks
DN must reserve space for a full block when an RBW block is created
---
Key: HDFS-6898
URL: https://issues.apache.org/jira/browse/HDFS-6898
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.5.0
Reporter: Gopal V
Assignee: Arpit Agarwal
Attachments: HDFS-6898.01.patch, HDFS-6898.03.patch
DN will successfully create two RBW blocks on the same volume even if the
free space is sufficient for just one full block.
One or both block writers may subsequently get a DiskOutOfSpace exception.
This can be avoided by allocating space up front.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6898) DN must reserve space for a full block when an RBW block is created
[
https://issues.apache.org/jira/browse/HDFS-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14109922#comment-14109922
]
Arpit Agarwal commented on HDFS-6898:
-
Thanks for looking at the patch.
I am not sure whether the Linux implementation of {{fallocate}} results in
extra file I/O. However we still have the other issue that Colin mentioned - we
cannot deduce the end of file on restart since our block file format lacks any
header/meta information.
For now we can just reserve space for a full block when an RBW block is
created. The reserved space is progressively reduced as bytes are ack'ed to
the client. Any remaining space is released when the block is finalized.
[~cmccabe], the attached .03.patch is current.
DN must reserve space for a full block when an RBW block is created
---
Key: HDFS-6898
URL: https://issues.apache.org/jira/browse/HDFS-6898
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.5.0
Reporter: Gopal V
Assignee: Arpit Agarwal
Attachments: HDFS-6898.01.patch, HDFS-6898.03.patch
DN will successfully create two RBW blocks on the same volume even if the
free space is sufficient for just one full block.
One or both block writers may subsequently get a DiskOutOfSpace exception.
This can be avoided by allocating space up front.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6898) DN must reserve space for a full block when an RBW block is created
[ https://issues.apache.org/jira/browse/HDFS-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14109966#comment-14109966 ] Colin Patrick McCabe commented on HDFS-6898: bq. Colin Patrick McCabe Will fallocate trigger disk I/O? I thought it only manipulates metadata: http://linux.die.net/man/1/fallocate – this is done quickly by allocating blocks and marking them as uninitialized, requiring no IO to the data blocks It depends on the underlying file system. For a filesystem which uses extents, like ext4, not much I/O would be required. For something like ext2, you have to allocate each ext2 block individually and link it to the rest of them, which could be quite a lot of I/O. That's a fair point, though... it will not be 128 MB of I/O in either case... it would be less than that. It's all metadata I/O essentially. bq. However we still have the other issue that Colin mentioned - we cannot deduce the end of file on restart since our block file format lacks any header/meta information. Yeah. bq. Colin Patrick McCabe, the attached .03.patch is current. Thanks. Will try to take a look later today DN must reserve space for a full block when an RBW block is created --- Key: HDFS-6898 URL: https://issues.apache.org/jira/browse/HDFS-6898 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 2.5.0 Reporter: Gopal V Assignee: Arpit Agarwal Attachments: HDFS-6898.01.patch, HDFS-6898.03.patch DN will successfully create two RBW blocks on the same volume even if the free space is sufficient for just one full block. One or both block writers may subsequently get a DiskOutOfSpace exception. This can be avoided by allocating space up front. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6898) DN must reserve space for a full block when an RBW block is created
[
https://issues.apache.org/jira/browse/HDFS-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14110133#comment-14110133
]
Colin Patrick McCabe commented on HDFS-6898:
{code}
/**
@@ -62,8 +62,8 @@ public ReplicaBeingWritten(Block block,
* @param writer a thread that is writing to this replica
*/
public ReplicaBeingWritten(long blockId, long len, long genStamp,
- FsVolumeSpi vol, File dir, Thread writer ) {
-super( blockId, len, genStamp, vol, dir, writer);
+ FsVolumeSpi vol, File dir, Thread writer, long maxBlockLengthHint) {
+super(blockId, len, genStamp, vol, dir, writer, maxBlockLengthHint);
{code}
Need JavaDoc for the new parameter, here and in a few other places.
{code}
+ long spacePreviouslyReserved = 0;
+ if (replicaInfo instanceof ReplicaBeingWritten) {
+spacePreviouslyReserved =
+((ReplicaBeingWritten) replicaInfo).getBytesReserved();
+ }
+ File dest = v.addFinalizedBlock(bpid, replicaInfo, f,
spacePreviouslyReserved);
{code}
instanceof seems ugly here. How about a method in the base class that returns
0, which gets overridden by ReplicaBeingWritten?
It seems like some places we talk about reserved space and others we use
block length hints. Maybe it makes more sense to use all one terminology or
the other, and have a comment in the code that the reserved space comes from
block length hints?
The unit test is good, but we should make absolutely sure that rbw space can't
leak. Can we add a stress test that creates and removes a bunch of files and
checks that we end up back at 0 reserved space?
DN must reserve space for a full block when an RBW block is created
---
Key: HDFS-6898
URL: https://issues.apache.org/jira/browse/HDFS-6898
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.5.0
Reporter: Gopal V
Assignee: Arpit Agarwal
Attachments: HDFS-6898.01.patch, HDFS-6898.03.patch
DN will successfully create two RBW blocks on the same volume even if the
free space is sufficient for just one full block.
One or both block writers may subsequently get a DiskOutOfSpace exception.
This can be avoided by allocating space up front.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6898) DN must reserve space for a full block when an RBW block is created
[
https://issues.apache.org/jira/browse/HDFS-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107937#comment-14107937
]
Hadoop QA commented on HDFS-6898:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12663482/HDFS-6898.03.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer
org.apache.hadoop.security.TestRefreshUserMappings
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7739//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7739//console
This message is automatically generated.
DN must reserve space for a full block when an RBW block is created
---
Key: HDFS-6898
URL: https://issues.apache.org/jira/browse/HDFS-6898
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.5.0
Reporter: Gopal V
Assignee: Arpit Agarwal
Attachments: HDFS-6898.01.patch, HDFS-6898.03.patch
DN will successfully create two RBW blocks on the same volume even if the
free space is sufficient for just one full block.
One or both block writers may subsequently get a DiskOutOfSpace exception.
This can be avoided by allocating space up front.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6898) DN must reserve space for a full block when an RBW block is created
[ https://issues.apache.org/jira/browse/HDFS-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107580#comment-14107580 ] Zhe Zhang commented on HDFS-6898: - A related question is whether DN should pre-allocate disk space for received RBW blocks. If the underlying local FS supports fallocate, this will improve the disk layout of the block. DN must reserve space for a full block when an RBW block is created --- Key: HDFS-6898 URL: https://issues.apache.org/jira/browse/HDFS-6898 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 2.5.0 Reporter: Gopal V Assignee: Arpit Agarwal Attachments: HDFS-6898.01.patch, HDFS-6898.03.patch DN will successfully create two RBW blocks on the same volume even if the free space is sufficient for just one full block. One or both block writers may subsequently get a DiskOutOfSpace exception. This can be avoided by allocating space up front. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6898) DN must reserve space for a full block when an RBW block is created
[
https://issues.apache.org/jira/browse/HDFS-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106008#comment-14106008
]
Hadoop QA commented on HDFS-6898:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12663482/HDFS-6898.03.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7702//console
This message is automatically generated.
DN must reserve space for a full block when an RBW block is created
---
Key: HDFS-6898
URL: https://issues.apache.org/jira/browse/HDFS-6898
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.5.0
Reporter: Gopal V
Assignee: Arpit Agarwal
Attachments: HDFS-6898.01.patch, HDFS-6898.03.patch
DN will successfully create two RBW blocks on the same volume even if the
free space is sufficient for just one full block.
One or both block writers may subsequently get a DiskOutOfSpace exception.
This can be avoided by allocating space up front.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6898) DN must reserve space for a full block when an RBW block is created
[ https://issues.apache.org/jira/browse/HDFS-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106041#comment-14106041 ] Arpit Agarwal commented on HDFS-6898: - Cancelled, patch depends on HDFS-6899. DN must reserve space for a full block when an RBW block is created --- Key: HDFS-6898 URL: https://issues.apache.org/jira/browse/HDFS-6898 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 2.5.0 Reporter: Gopal V Assignee: Arpit Agarwal Attachments: HDFS-6898.01.patch, HDFS-6898.03.patch DN will successfully create two RBW blocks on the same volume even if the free space is sufficient for just one full block. One or both block writers may subsequently get a DiskOutOfSpace exception. This can be avoided by allocating space up front. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6898) DN must reserve space for a full block when an RBW block is created
[ https://issues.apache.org/jira/browse/HDFS-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14104835#comment-14104835 ] Arpit Agarwal commented on HDFS-6898: - Reported by [~gopalv]. DN must reserve space for a full block when an RBW block is created --- Key: HDFS-6898 URL: https://issues.apache.org/jira/browse/HDFS-6898 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 2.5.0 Reporter: Gopal V Assignee: Arpit Agarwal DN will successfully create two RBW blocks on the same volume even if the free space is sufficient for just one full block. One or both block writers may subsequently get a DiskOutOfSpace exception. This can be avoided by allocating space up front. -- This message was sent by Atlassian JIRA (v6.2#6252)
