[jira] [Commented] (HDFS-10271) Extra bytes are getting released from reservedSpace for append

2016-04-12 Thread Vinayakumar B (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-10271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236855#comment-15236855
 ] 

Vinayakumar B commented on HDFS-10271:
--

I think brahma has replied for those questions in HDFS-9530 
[here|https://issues.apache.org/jira/browse/HDFS-9530?focusedCommentId=15231505=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15231505].
 
Perhaps, this issue is straight forward, and related to append.
So we shall continue that discussion in HDFS-9530 itself.


> Extra bytes are getting released from reservedSpace for append
> --
>
> Key: HDFS-10271
> URL: https://issues.apache.org/jira/browse/HDFS-10271
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Critical
> Fix For: 2.7.3, 2.6.5
>
> Attachments: HDFS-10271-01.patch, HDFS-10271-branch-2.7-01.patch
>
>
> 1. File already have some bytes available in block. (ex: 1024B)
> 2. Re-open the file for append, (Here reserving for (BlockSize-1024) bytes)
> 3. write one byte and flush, 
> 4. close()
> After close(), releasing *BlockSize-1* bytes from reservedspace instead of 
> *BlockSize-1025* bytes.
> Extra bytes released from reservedSpace may create problem for other writers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-10271) Extra bytes are getting released from reservedSpace for append

2016-04-12 Thread Ravi Prakash (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-10271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236780#comment-15236780
 ] 

Ravi Prakash commented on HDFS-10271:
-

Here are the other places that I thought reserved space should have been 
modified : 
https://issues.apache.org/jira/browse/HDFS-9530?focusedCommentId=15231164=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15231164
 . 

> Extra bytes are getting released from reservedSpace for append
> --
>
> Key: HDFS-10271
> URL: https://issues.apache.org/jira/browse/HDFS-10271
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Critical
> Fix For: 2.7.3, 2.6.5
>
> Attachments: HDFS-10271-01.patch, HDFS-10271-branch-2.7-01.patch
>
>
> 1. File already have some bytes available in block. (ex: 1024B)
> 2. Re-open the file for append, (Here reserving for (BlockSize-1024) bytes)
> 3. write one byte and flush, 
> 4. close()
> After close(), releasing *BlockSize-1* bytes from reservedspace instead of 
> *BlockSize-1025* bytes.
> Extra bytes released from reservedSpace may create problem for other writers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-10271) Extra bytes are getting released from reservedSpace for append

2016-04-12 Thread Brahma Reddy Battula (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-10271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236691#comment-15236691
 ] 

Brahma Reddy Battula commented on HDFS-10271:
-

[~vinayrpet] thanks a lot for review and commit..

> Extra bytes are getting released from reservedSpace for append
> --
>
> Key: HDFS-10271
> URL: https://issues.apache.org/jira/browse/HDFS-10271
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Critical
> Fix For: 2.7.3, 2.6.5
>
> Attachments: HDFS-10271-01.patch, HDFS-10271-branch-2.7-01.patch
>
>
> 1. File already have some bytes available in block. (ex: 1024B)
> 2. Re-open the file for append, (Here reserving for (BlockSize-1024) bytes)
> 3. write one byte and flush, 
> 4. close()
> After close(), releasing *BlockSize-1* bytes from reservedspace instead of 
> *BlockSize-1025* bytes.
> Extra bytes released from reservedSpace may create problem for other writers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-10271) Extra bytes are getting released from reservedSpace for append

2016-04-12 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-10271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236682#comment-15236682
 ] 

Hudson commented on HDFS-10271:
---

FAILURE: Integrated in Hadoop-trunk-Commit #9596 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/9596/])
HDFS-10271. Extra bytes are getting released from reservedSpace for 
(vinayakumarb: rev a9a607f8fc0d996af3fb37f7efa7591d6655900d)
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestSpaceReservation.java


> Extra bytes are getting released from reservedSpace for append
> --
>
> Key: HDFS-10271
> URL: https://issues.apache.org/jira/browse/HDFS-10271
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Critical
> Fix For: 2.7.3, 2.6.5
>
> Attachments: HDFS-10271-01.patch, HDFS-10271-branch-2.7-01.patch
>
>
> 1. File already have some bytes available in block. (ex: 1024B)
> 2. Re-open the file for append, (Here reserving for (BlockSize-1024) bytes)
> 3. write one byte and flush, 
> 4. close()
> After close(), releasing *BlockSize-1* bytes from reservedspace instead of 
> *BlockSize-1025* bytes.
> Extra bytes released from reservedSpace may create problem for other writers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-10271) Extra bytes are getting released from reservedSpace for append

2016-04-11 Thread Vinayakumar B (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-10271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236627#comment-15236627
 ] 

Vinayakumar B commented on HDFS-10271:
--

This seems to be a pretty straight-forward bug.
Simple Append op releases more reserved space than it actually should.

Thanks [~brahmareddy] for nice find.
+1 for the patch.
I will commit it shortly.


> Extra bytes are getting released from reservedSpace for append
> --
>
> Key: HDFS-10271
> URL: https://issues.apache.org/jira/browse/HDFS-10271
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Critical
> Attachments: HDFS-10271-01.patch, HDFS-10271-branch-2.7-01.patch
>
>
> 1. File already have some bytes available in block. (ex: 1024B)
> 2. Re-open the file for append, (Here reserving for (BlockSize-1024) bytes)
> 3. write one byte and flush, 
> 4. close()
> After close(), releasing *BlockSize-1* bytes from reservedspace instead of 
> *BlockSize-1025* bytes.
> Extra bytes reserved may create problem for other writers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-10271) Extra bytes are getting released from reservedSpace for append

2016-04-11 Thread Ravi Prakash (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-10271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236418#comment-15236418
 ] 

Ravi Prakash commented on HDFS-10271:
-

Hi Vinod!

Counting of reservedSpace on Datanodes is all messed up right now :( I don't 
know if you should block on getting all these fixes in, because they may be a 
while. 

These issues already exist in 2.7.* (and I haven't heard very many complaints). 
There exists a workaround (restart the Datanode). 

I'm going to push out Target Version to 2.7.4 . Please revert if you disagree

> Extra bytes are getting released from reservedSpace for append
> --
>
> Key: HDFS-10271
> URL: https://issues.apache.org/jira/browse/HDFS-10271
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Critical
> Attachments: HDFS-10271-01.patch, HDFS-10271-branch-2.7-01.patch
>
>
> 1. File already have some bytes available in block. (ex: 1024B)
> 2. Re-open the file for append, (Here reserving for (BlockSize-1024) bytes)
> 3. write one byte and flush, 
> 4. close()
> After close(), releasing *BlockSize-1* bytes from reservedspace instead of 
> *BlockSize-1025* bytes.
> Extra bytes reserved may create problem for other writers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-10271) Extra bytes are getting released from reservedSpace for append

2016-04-11 Thread Vinod Kumar Vavilapalli (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-10271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15235758#comment-15235758
 ] 

Vinod Kumar Vavilapalli commented on HDFS-10271:


Can somebody from HDFS please help review / commit this? It's one of the last 
few blocking tickets for 2.7.3!

> Extra bytes are getting released from reservedSpace for append
> --
>
> Key: HDFS-10271
> URL: https://issues.apache.org/jira/browse/HDFS-10271
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Critical
> Attachments: HDFS-10271-01.patch, HDFS-10271-branch-2.7-01.patch
>
>
> 1. File already have some bytes available in block. (ex: 1024B)
> 2. Re-open the file for append, (Here reserving for (BlockSize-1024) bytes)
> 3. write one byte and flush, 
> 4. close()
> After close(), releasing *BlockSize-1* bytes from reservedspace instead of 
> *BlockSize-1025* bytes.
> Extra bytes reserved may create problem for other writers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-10271) Extra bytes are getting released from reservedSpace for append

2016-04-08 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-10271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15232079#comment-15232079
 ] 

Hadoop QA commented on HDFS-10271:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m 
5s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 33s 
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 13s 
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
37s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 24s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
23s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
47s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 58s 
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 8s 
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 28s 
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 28s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 3s 
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 3s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
30s {color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: patch generated 0 
new + 121 unchanged - 1 fixed = 121 total (was 122) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 21s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 7s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 52s 
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 4s 
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 124m 6s {color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_77. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 82m 48s {color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m 
4s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 250m 10s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_77 Failed junit tests | 
hadoop.hdfs.server.datanode.TestDirectoryScanner |
|   | hadoop.hdfs.server.namenode.ha.TestHAAppend |
|   | hadoop.hdfs.qjournal.TestSecureNNWithQJM |
|   | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
|   | hadoop.hdfs.server.datanode.TestDataNodeUUID |
|   | hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot |
|   | hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations |
|   | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
|   | hadoop.hdfs.TestFileAppend |
|   | hadoop.hdfs.TestErasureCodeBenchmarkThroughput |
|

[jira] [Commented] (HDFS-10271) Extra bytes are getting released from reservedSpace for append

2016-04-08 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-10271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15232018#comment-15232018
 ] 

Hadoop QA commented on HDFS-10271:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 9m 20s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 
5s {color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 18s 
{color} | {color:green} branch-2.7 passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 9s 
{color} | {color:green} branch-2.7 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
28s {color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 6s 
{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
13s {color} | {color:green} branch-2.7 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m 14s 
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in branch-2.7 has 1 
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 22s 
{color} | {color:green} branch-2.7 passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 6s 
{color} | {color:green} branch-2.7 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
4s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 14s 
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 14s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 3s 
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 3s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 25s 
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs: patch generated 1 new + 
127 unchanged - 2 fixed = 128 total (was 129) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 2s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
11s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 1s 
{color} | {color:red} The patch has 1200 line(s) that end in whitespace. Use 
git apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 29s 
{color} | {color:red} The patch has 246 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 21s 
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 8s 
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 68m 5s {color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_77. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 63m 46s {color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 1m 6s 
{color} | {color:red} Patch generated 65 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 175m 36s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_77 Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
|   | hadoop.hdfs.server.datanode.TestBlockScanner |
|   | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
|   | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
|   | hadoop.hdfs.qjournal.TestSecureNNWithQJM |
| JDK v1.7.0_95 Failed