date:20150224

[jira] [Updated] (HDFS-7834) Allow HDFS to bind to ipv6 conditionally

2015-02-24 Thread Allen Wittenauer (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-7834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer updated HDFS-7834:
---
Component/s: scripts

 Allow HDFS to bind to ipv6 conditionally
 

 Key: HDFS-7834
 URL: https://issues.apache.org/jira/browse/HDFS-7834
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: scripts
Affects Versions: 2.6.0
Reporter: Elliott Clark
Assignee: Elliott Clark
 Attachments: HDFS-7834-branch-2-0.patch


 Currently the bash scripts unconditionally add -Djava.net.preferIPv4Stack=true
 While this was needed a while ago. IPV6 on java works much better now and 
 there should be a way to allow it to bind dual stack if needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HDFS-7834) Allow HDFS to bind to ipv6 conditionally

2015-02-24 Thread Allen Wittenauer (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-7834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14335423#comment-14335423
 ] 

Allen Wittenauer edited comment on HDFS-7834 at 2/24/15 9:00 PM:
-

In trunk, you can set HADOOP_OPTS to something (blank, for example) and set 
HADOOP_ALLOW_IPV6 to yes.  


was (Author: aw):
In trunk, you can set HADOOP_ALLOW_IPV6. 

 Allow HDFS to bind to ipv6 conditionally
 

 Key: HDFS-7834
 URL: https://issues.apache.org/jira/browse/HDFS-7834
 Project: Hadoop HDFS
  Issue Type: Improvement
Affects Versions: 2.6.0
Reporter: Elliott Clark
Assignee: Elliott Clark
 Fix For: 2.7.0


 Currently the bash scripts unconditionally add -Djava.net.preferIPv4Stack=true
 While this was needed a while ago. IPV6 on java works much better now and 
 there should be a way to allow it to bind dual stack if needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-7834) Allow HDFS to bind to ipv6 conditionally

2015-02-24 Thread Allen Wittenauer (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-7834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer updated HDFS-7834:
---
Fix Version/s: (was: 2.7.0)

 Allow HDFS to bind to ipv6 conditionally
 

 Key: HDFS-7834
 URL: https://issues.apache.org/jira/browse/HDFS-7834
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: scripts
Affects Versions: 2.6.0
Reporter: Elliott Clark
Assignee: Elliott Clark
 Attachments: HDFS-7834-branch-2-0.patch


 Currently the bash scripts unconditionally add -Djava.net.preferIPv4Stack=true
 While this was needed a while ago. IPV6 on java works much better now and 
 there should be a way to allow it to bind dual stack if needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-7834) Allow HDFS to bind to ipv6 conditionally

2015-02-24 Thread Elliott Clark (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-7834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HDFS-7834:

Attachment: HDFS-7834-branch-2-0.patch

Here's a patch for branch-2.

 Allow HDFS to bind to ipv6 conditionally
 

 Key: HDFS-7834
 URL: https://issues.apache.org/jira/browse/HDFS-7834
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: scripts
Affects Versions: 2.6.0
Reporter: Elliott Clark
Assignee: Elliott Clark
 Attachments: HDFS-7834-branch-2-0.patch


 Currently the bash scripts unconditionally add -Djava.net.preferIPv4Stack=true
 While this was needed a while ago. IPV6 on java works much better now and 
 there should be a way to allow it to bind dual stack if needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7818) DataNode throws NPE if the WebHdfs URL does not contain the offset parameter

2015-02-24 Thread Eric Payne (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-7818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14335483#comment-14335483
 ] 

Eric Payne commented on HDFS-7818:
--

Now that I look at it, the patch in HDFS-7818.v3.txt is not exactly correct 
either. I think that if we want to keep the NULL check in a constructor, it 
should be done in {{OffsetParam(final Long value)}} instead of 
{{OffsetParam(final String str)}}, since the latter invokes the former.

 DataNode throws NPE if the WebHdfs URL does not contain the offset parameter
 

 Key: HDFS-7818
 URL: https://issues.apache.org/jira/browse/HDFS-7818
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Affects Versions: 2.7.0
Reporter: Eric Payne
Assignee: Eric Payne
 Attachments: HDFS-7818.v1.txt, HDFS-7818.v2.txt, HDFS-7818.v3.txt


 This is a regression in 2.7 and later.
 {{hadoop fs -cat}} over webhdfs works, but {{hadoop fs -text}} does not:
 {code}
 $ hadoop fs -cat webhdfs://myhost.com/tmp/test.1
 ... output ...
 $ hadoop fs -text webhdfs://myhost.com/tmp/test.1
 text: org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): 
 null
   at 
 org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:165)
   at 
 org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:358)
   at 
 org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:91)
   at 
 org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:615)
   at 
 org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:463)
   at 
 org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:492)
 ...
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7834) Allow HDFS to bind to ipv6 conditionally

2015-02-24 Thread Allen Wittenauer (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-7834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14335423#comment-14335423
 ] 

Allen Wittenauer commented on HDFS-7834:


In trunk, you can set HADOOP_ALLOW_IPV6. 

 Allow HDFS to bind to ipv6 conditionally
 

 Key: HDFS-7834
 URL: https://issues.apache.org/jira/browse/HDFS-7834
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Elliott Clark
Assignee: Elliott Clark

 Currently the bash scripts unconditionally add -Djava.net.preferIPv4Stack=true
 While this was needed a while ago. IPV6 on java works much better now and 
 there should be a way to allow it to bind dual stack if needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-7411) Refactor and improve decommissioning logic into DecommissionManager

2015-02-24 Thread Andrew Wang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-7411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang updated HDFS-7411:
--
Attachment: hdfs-7411.011.patch

Sorry for the delay on this everyone, I was on vacation last week.

I've implemented Chris D's suggestion (with unit test) that does a per node 
limit. If the deprecated config key is set, it is used preferentially over the 
default for the new config key.

Nicholas, does this satisfy your criteria?

 Refactor and improve decommissioning logic into DecommissionManager
 ---

 Key: HDFS-7411
 URL: https://issues.apache.org/jira/browse/HDFS-7411
 Project: Hadoop HDFS
  Issue Type: Improvement
Affects Versions: 2.5.1
Reporter: Andrew Wang
Assignee: Andrew Wang
 Attachments: hdfs-7411.001.patch, hdfs-7411.002.patch, 
 hdfs-7411.003.patch, hdfs-7411.004.patch, hdfs-7411.005.patch, 
 hdfs-7411.006.patch, hdfs-7411.007.patch, hdfs-7411.008.patch, 
 hdfs-7411.009.patch, hdfs-7411.010.patch, hdfs-7411.011.patch


 Would be nice to split out decommission logic from DatanodeManager to 
 DecommissionManager.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-6962) ACLs inheritance conflict with umaskmode

2015-02-24 Thread Chris Nauroth (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Nauroth updated HDFS-6962:

Target Version/s: 3.0.0  (was: 2.7.0)

Hello, [~usrikanth].  Thank you for posting a prototype patch and providing a 
great written summary.

I'm now certain that it's impossible to make this change in a 
backwards-compatible way in the 2.x line.  The biggest challenge is what 
happens if someone upgrades the client ahead of the NameNode.  In that case, 
neither the client nor the NameNode would apply the umask.  Effectively, that 
means the upgraded client would start creating directories with 777 and files 
with 666, which of course would compromise security.

Another potential issue is that existing users may be accustomed to the 
behavior of the current implementation, despite this deviation from the POSIX 
ACL spec.  The effect of the proposed change would be to widen access, because 
it would stop applying umask in certain cases.  Users might find it surprising 
if their default ACLs stopped restricting access after an upgrade, and some 
would argue that this is a form of incompatibility with existing persistent 
data (metadata).  This is always a fine line, but I do suspect some would see 
it as an incompatibility.

I'm retargeting this to 3.0.0.  That means we'll also have the option of 
creating a much simpler patch, because we'll have freedom to make 
backwards-incompatible changes.

Here are a few notes on the prototype patch, although I suspect it will go in a 
very different direction for 3.0.0 anyway.
# {{CommandWithDestination}}: This change also probably would have constituted 
a backwards incompatibility.  Prior versions create files as 666 filtered by 
{{fs.permissions.umask-mode}}, not based on the permissions from the source 
file system.  I see from your notes that you were aiming to replicate the 
behavior you saw on Linux.  It might be worthwhile for us to consider doing 
that for consistency with other file systems, but it would be 
backwards-incompatible in 2.x.
# {{FSDirectory}}: Here, the NameNode is applying umask based on its configured 
value for {{fs.permissions.umask-mode}}.  Unfortunately, this won't work in the 
general case, because it's not guaranteed that the client and the NameNode are 
running with the same set of configuration files.  They might have different 
values configured for {{fs.permissions.umask-mode}}, or the client might have 
overridden it with a -D option on the command line.

 ACLs inheritance conflict with umaskmode
 

 Key: HDFS-6962
 URL: https://issues.apache.org/jira/browse/HDFS-6962
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: security
Affects Versions: 2.4.1
 Environment: CentOS release 6.5 (Final)
Reporter: LINTE
Assignee: Srikanth Upputuri
  Labels: hadoop, security
 Attachments: HDFS-6962.1.patch


 In hdfs-site.xml 
 property
 namedfs.umaskmode/name
 value027/value
 /property
 1/ Create a directory as superuser
 bash# hdfs dfs -mkdir  /tmp/ACLS
 2/ set default ACLs on this directory rwx access for group readwrite and user 
 toto
 bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS
 bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS
 3/ check ACLs /tmp/ACLS/
 bash# hdfs dfs -getfacl /tmp/ACLS/
 # file: /tmp/ACLS
 # owner: hdfs
 # group: hadoop
 user::rwx
 group::r-x
 other::---
 default:user::rwx
 default:user:toto:rwx
 default:group::r-x
 default:group:readwrite:rwx
 default:mask::rwx
 default:other::---
 user::rwx | group::r-x | other::--- matches with the umaskmode defined in 
 hdfs-site.xml, everything ok !
 default:group:readwrite:rwx allow readwrite group with rwx access for 
 inhéritance.
 default:user:toto:rwx allow toto user with rwx access for inhéritance.
 default:mask::rwx inhéritance mask is rwx, so no mask
 4/ Create a subdir to test inheritance of ACL
 bash# hdfs dfs -mkdir  /tmp/ACLS/hdfs
 5/ check ACLs /tmp/ACLS/hdfs
 bash# hdfs dfs -getfacl /tmp/ACLS/hdfs
 # file: /tmp/ACLS/hdfs
 # owner: hdfs
 # group: hadoop
 user::rwx
 user:toto:rwx   #effective:r-x
 group::r-x
 group:readwrite:rwx #effective:r-x
 mask::r-x
 other::---
 default:user::rwx
 default:user:toto:rwx
 default:group::r-x
 default:group:readwrite:rwx
 default:mask::rwx
 default:other::---
 Here we can see that the readwrite group has rwx ACL bu only r-x is effective 
 because the mask is r-x (mask::r-x) in spite of default mask for inheritance 
 is set to default:mask::rwx on /tmp/ACLS/
 6/ Modifiy hdfs-site.xml et restart namenode
 property
 namedfs.umaskmode/name
 value010/value
 /property
 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode
 bash# hdfs dfs -mkdir  /tmp/ACLS/hdfs2
 8/ Check ACL on /tmp/ACLS/hdfs2
 bash# hdfs dfs -getfacl

[jira] [Commented] (HDFS-7818) DataNode throws NPE if the WebHdfs URL does not contain the offset parameter

2015-02-24 Thread Eric Payne (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-7818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14335438#comment-14335438
 ] 

Eric Payne commented on HDFS-7818:
--

Thank you for your review, [~wheat9]
bq. Maybe it might make more sense to introduce a new method {{getOffset()}} in 
{{OffsetParam}}.
If a {{getOffset()}} method is created instead of handling the NULL case in the 
constructor as is done in the HDFS-7818.V3.txt patch, won't I also have to 
change all of the {{offset.getValue()}} calls to {{offset.getOffset()}} in the 
{{NamenodeWebHdfsMethods}} class?

The change in the current patch seems less risky because it catches the NULL 
case during construction of the object and has less code change.

 DataNode throws NPE if the WebHdfs URL does not contain the offset parameter
 

 Key: HDFS-7818
 URL: https://issues.apache.org/jira/browse/HDFS-7818
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Affects Versions: 2.7.0
Reporter: Eric Payne
Assignee: Eric Payne
 Attachments: HDFS-7818.v1.txt, HDFS-7818.v2.txt, HDFS-7818.v3.txt


 This is a regression in 2.7 and later.
 {{hadoop fs -cat}} over webhdfs works, but {{hadoop fs -text}} does not:
 {code}
 $ hadoop fs -cat webhdfs://myhost.com/tmp/test.1
 ... output ...
 $ hadoop fs -text webhdfs://myhost.com/tmp/test.1
 text: org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): 
 null
   at 
 org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:165)
   at 
 org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:358)
   at 
 org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:91)
   at 
 org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:615)
   at 
 org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:463)
   at 
 org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:492)
 ...
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-778) DistributedFileSystem.getFileBlockLocations() may occasionally return numeric ips as hostnames.

2015-02-24 Thread Allen Wittenauer (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer updated HDFS-778:
--
Labels: ipv6  (was: )

 DistributedFileSystem.getFileBlockLocations() may occasionally return numeric 
 ips as hostnames.
 ---

 Key: HDFS-778
 URL: https://issues.apache.org/jira/browse/HDFS-778
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Hong Tang
  Labels: ipv6

 DistributedFileSystem.getFileBlockLocations() may occasionally return numeric 
 ips as hostnames. This seems to be a breach of the 
 FileSystem.getFileBlockLocation() contract:
 {noformat}
   /**
* Return an array containing hostnames, offset and size of 
* portions of the given file.  For a nonexistent 
* file or regions, null will be returned.
*
* This call is most helpful with DFS, where it returns 
* hostnames of machines that contain the given file.
*
* The FileSystem will simply return an elt containing 'localhost'.
*/
   public BlockLocation[] getFileBlockLocations(FileStatus file, 
   long start, long len) throws IOException
 {noformat}
 One (maybe minor) consequence of this issue is: When a job includes such 
 numeric ips in in its splits' locations, JobTracker would not be able to 
 assign the job's map tasks local to the file blocks.
 We should either fix the implementation or change the contract. In the latter 
 case, JobTracker needs to be fixed to maintain both the hostnames and ips of 
 the TaskTrackers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-7835) make initial sleeptime in locateFollowingBlock configurable for DFSClient.

2015-02-24 Thread zhihai xu (JIRA)

zhihai xu created HDFS-7835:
---

 Summary: make initial sleeptime in locateFollowingBlock 
configurable for DFSClient.
 Key: HDFS-7835
 URL: https://issues.apache.org/jira/browse/HDFS-7835
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: dfsclient
Reporter: zhihai xu
Assignee: zhihai xu


Make initial sleeptime in locateFollowingBlock configurable for DFSClient.
Current the sleeptime/localTimeout in locateFollowingBlock/completeFile from 
DFSOutputStream is hard-coded as 400 ms, but retries can be configured by 
dfs.client.block.write.locateFollowingBlock.retries. We should also make the 
initial sleeptime configurable to give user more flexibility to control both 
retry and delay.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-7835) make initial sleeptime in locateFollowingBlock configurable for DFSClient.

2015-02-24 Thread zhihai xu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-7835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhihai xu updated HDFS-7835:

Status: Patch Available  (was: Open)

 make initial sleeptime in locateFollowingBlock configurable for DFSClient.
 --

 Key: HDFS-7835
 URL: https://issues.apache.org/jira/browse/HDFS-7835
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: dfsclient
Reporter: zhihai xu
Assignee: zhihai xu
 Attachments: HDFS-7835.000.patch


 Make initial sleeptime in locateFollowingBlock configurable for DFSClient.
 Current the sleeptime/localTimeout in locateFollowingBlock/completeFile from 
 DFSOutputStream is hard-coded as 400 ms, but retries can be configured by 
 dfs.client.block.write.locateFollowingBlock.retries. We should also make 
 the initial sleeptime configurable to give user more flexibility to control 
 both retry and delay.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-7835) make initial sleeptime in locateFollowingBlock configurable for DFSClient.

2015-02-24 Thread zhihai xu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-7835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhihai xu updated HDFS-7835:

Attachment: HDFS-7835.000.patch

 make initial sleeptime in locateFollowingBlock configurable for DFSClient.
 --

 Key: HDFS-7835
 URL: https://issues.apache.org/jira/browse/HDFS-7835
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: dfsclient
Reporter: zhihai xu
Assignee: zhihai xu
 Attachments: HDFS-7835.000.patch


 Make initial sleeptime in locateFollowingBlock configurable for DFSClient.
 Current the sleeptime/localTimeout in locateFollowingBlock/completeFile from 
 DFSOutputStream is hard-coded as 400 ms, but retries can be configured by 
 dfs.client.block.write.locateFollowingBlock.retries. We should also make 
 the initial sleeptime configurable to give user more flexibility to control 
 both retry and delay.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7537) fsck is confusing when dfs.namenode.replication.min 1 missing replicas NN restart

2015-02-24 Thread Allen Wittenauer (JIRA)

[
https://issues.apache.org/jira/browse/HDFS-7537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14335523#comment-14335523
]

Allen Wittenauer commented on HDFS-7537:

bq. When numUnderMinimalRelicatedBlocks 0 and there is no missing/corrupted
block, all under minimal replicated blocks have at least one good replica so
that they can be replicated and there is no data loss. It makes sense to
consider the file system as healthy.

Exactly this.

I made a prototype to play with. One of things I did was put the number of
blocks that didn't meet the replication minimum surrounded by the asterisks
that the corrupted output did. This made it absolutely crystal clear why the
NN wasn't coming out of safemode.

fsck is confusing when dfs.namenode.replication.min 1 missing replicas
NN restart
-

Key: HDFS-7537
URL: https://issues.apache.org/jira/browse/HDFS-7537
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Reporter: Allen Wittenauer
Assignee: GAO Rui
Attachments: HDFS-7537.1.patch, dfs-min-2-fsck.png, dfs-min-2.png

If minimum replication is set to 2 or higher and some of those replicas are
missing and the namenode restarts, it isn't always obvious that the missing
replicas are the reason why the namenode isn't leaving safemode. We should
improve the output of fsck and the web UI to make it obvious that the missing
blocks are from unmet replicas vs. completely/totally missing.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7280) Use netty 4 in WebImageViewer

2015-02-24 Thread Yongjun Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-7280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14335365#comment-14335365
 ] 

Yongjun Zhang commented on HDFS-7280:
-

Hi [~wheat9],

Thanks for your work on this jira. I have some questions, in general, what 
impact it is if there is any on user side when we change from using netty3 to 
netty4? Anything special users need to do? any compatibility issue with tools 
that interface with hadoop?  Thanks.





 Use netty 4 in WebImageViewer
 -

 Key: HDFS-7280
 URL: https://issues.apache.org/jira/browse/HDFS-7280
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Haohui Mai
 Fix For: 2.7.0

 Attachments: HDFS-7280.000.patch, HDFS-7280.001.patch, 
 HDFS-7280.002.patch, HDFS-7280.003.patch, HDFS-7280.004.patch


 This jira changes WebImageViewer to use netty 4 instead of netty 3.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-7834) Allow HDFS to bind to ipv6 conditionally

2015-02-24 Thread Elliott Clark (JIRA)

Elliott Clark created HDFS-7834:
---

 Summary: Allow HDFS to bind to ipv6 conditionally
 Key: HDFS-7834
 URL: https://issues.apache.org/jira/browse/HDFS-7834
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Elliott Clark
Assignee: Elliott Clark


Currently the bash scripts unconditionally add -Djava.net.preferIPv4Stack=true

While this was needed a while ago. IPV6 on java works much better now and there 
should be a way to allow it to bind dual stack if needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-7834) Allow HDFS to bind to ipv6 conditionally

2015-02-24 Thread Elliott Clark (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-7834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HDFS-7834:

Affects Version/s: 2.6.0
Fix Version/s: 2.7.0

 Allow HDFS to bind to ipv6 conditionally
 

 Key: HDFS-7834
 URL: https://issues.apache.org/jira/browse/HDFS-7834
 Project: Hadoop HDFS
  Issue Type: Improvement
Affects Versions: 2.6.0
Reporter: Elliott Clark
Assignee: Elliott Clark
 Fix For: 2.7.0


 Currently the bash scripts unconditionally add -Djava.net.preferIPv4Stack=true
 While this was needed a while ago. IPV6 on java works much better now and 
 there should be a way to allow it to bind dual stack if needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7302) namenode -rollingUpgrade downgrade may finalize a rolling upgrade

2015-02-24 Thread Kai Sasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-7302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14334594#comment-14334594
 ] 

Kai Sasaki commented on HDFS-7302:
--

[~szetszwo] I may have some misunderstanding. I found there were some 
dependencies on FSImage or FSNamesystem and so on. Can I remove all 
dependencies? This downgrade option is not also necessary to these internal 
classes?

 namenode -rollingUpgrade downgrade may finalize a rolling upgrade
 -

 Key: HDFS-7302
 URL: https://issues.apache.org/jira/browse/HDFS-7302
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Reporter: Tsz Wo Nicholas Sze
Assignee: Kai Sasaki
  Labels: document, hdfs
 Attachments: HADOOP-7302.1.patch


 The namenode startup option -rollingUpgrade downgrade is originally 
 designed for downgrading cluster.  However, running namenode -rollingUpgrade 
 downgrade with the new software could result in finalizing the ongoing 
 rolling upgrade.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7537) fsck is confusing when dfs.namenode.replication.min 1 missing replicas NN restart

2015-02-24 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-7537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14334686#comment-14334686
 ] 

Hadoop QA commented on HDFS-7537:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12700346/HDFS-7537.1.patch
  against trunk revision 1dba572.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-hdfs-project/hadoop-hdfs.

Test results: 
https://builds.apache.org/job/PreCommit-HDFS-Build/9653//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9653//console

This message is automatically generated.

 fsck is confusing when dfs.namenode.replication.min  1  missing replicas 
  NN restart
 -

 Key: HDFS-7537
 URL: https://issues.apache.org/jira/browse/HDFS-7537
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: namenode
Reporter: Allen Wittenauer
Assignee: GAO Rui
 Attachments: HDFS-7537.1.patch, dfs-min-2-fsck.png, dfs-min-2.png


 If minimum replication is set to 2 or higher and some of those replicas are 
 missing and the namenode restarts, it isn't always obvious that the missing 
 replicas are the reason why the namenode isn't leaving safemode.  We should 
 improve the output of fsck and the web UI to make it obvious that the missing 
 blocks are from unmet replicas vs. completely/totally missing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7439) Add BlockOpResponseProto's message to DFSClient's exception message

2015-02-24 Thread Takanobu Asanuma (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-7439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14334703#comment-14334703
 ] 

Takanobu Asanuma commented on HDFS-7439:


Excuse me, how can I rebuild for this patch? I can’t log in Jenkins WebUI.

 Add BlockOpResponseProto's message to DFSClient's exception message
 ---

 Key: HDFS-7439
 URL: https://issues.apache.org/jira/browse/HDFS-7439
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Ming Ma
Assignee: Takanobu Asanuma
Priority: Minor
 Attachments: HDFS-7439.1.patch


 When (BlockOpResponseProto#getStatus() != SUCCESS), it helps with debugging 
 if DFSClient can add BlockOpResponseProto's message to the exception message 
 applications will get. For example, instead of
 {noformat}
 throw new IOException(Got error for OP_READ_BLOCK, self=
 + peer.getLocalAddressString() + , remote=
 + peer.getRemoteAddressString() + , for file  + file
 + , for pool  + block.getBlockPoolId() +  block  
 + block.getBlockId() + _ + block.getGenerationStamp());
 {noformat}
 It could be,
 {noformat}
 throw new IOException(Got error for OP_READ_BLOCK, self=
 + peer.getLocalAddressString() + , remote=
 + peer.getRemoteAddressString() + , for file  + file
 + , for pool  + block.getBlockPoolId() +  block  
 + block.getBlockId() + _ + block.getGenerationStamp()
 + , status message  + status.getMessage());
 {noformat}
 We might want to check out all the references to BlockOpResponseProto in 
 DFSClient.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7537) fsck is confusing when dfs.namenode.replication.min 1 missing replicas NN restart

2015-02-24 Thread GAO Rui (JIRA)

[
https://issues.apache.org/jira/browse/HDFS-7537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14334742#comment-14334742
]

GAO Rui commented on HDFS-7537:
---

Thank you very much for your review and comment.
1. I think minReplication may get its value from
DFSConfigKeys.DFS_NAMENODE_REPLICATION_MIN_KEY in the first place. I’ll try to
figure this out and add it to the output.
2. In Allen’s comment, the Mock-up output shows status as HEALTHY when
numUnderMinimalRelicatedBlocks 0. It’s his careless mistake or maybe he has
his reason to keep the status as HEALTHY while show the
numUnderMinimalRelicatedBlocks in the same time?
3. I haven’t added unit test before, but I’ll try to do that.
4. Sorry, I’ll fix it and avoid this kind of mistakes in future codes.

fsck is confusing when dfs.namenode.replication.min 1 missing replicas
NN restart
-

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7056) Snapshot support for truncate

2015-02-24 Thread Konstantin Shvachko (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-7056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14334562#comment-14334562
 ] 

Konstantin Shvachko commented on HDFS-7056:
---

The patch is up for review in HDFS-7831.

 Snapshot support for truncate
 -

 Key: HDFS-7056
 URL: https://issues.apache.org/jira/browse/HDFS-7056
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Affects Versions: 3.0.0
Reporter: Konstantin Shvachko
Assignee: Plamen Jeliazkov
 Fix For: 2.7.0

 Attachments: HDFS-3107-HDFS-7056-combined-13.patch, 
 HDFS-3107-HDFS-7056-combined-15.patch, HDFS-3107-HDFS-7056-combined.patch, 
 HDFS-3107-HDFS-7056-combined.patch, HDFS-3107-HDFS-7056-combined.patch, 
 HDFS-3107-HDFS-7056-combined.patch, HDFS-3107-HDFS-7056-combined.patch, 
 HDFS-3107-HDFS-7056-combined.patch, HDFS-3107-HDFS-7056-combined.patch, 
 HDFS-3107-HDFS-7056-combined.patch, HDFS-3107-HDFS-7056-combined.patch, 
 HDFS-7056-13.patch, HDFS-7056-15.patch, HDFS-7056.15_branch2.patch, 
 HDFS-7056.patch, HDFS-7056.patch, HDFS-7056.patch, HDFS-7056.patch, 
 HDFS-7056.patch, HDFS-7056.patch, HDFS-7056.patch, HDFS-7056.patch, 
 HDFSSnapshotWithTruncateDesign.docx, HDFSSnapshotWithTruncateDesign.docx, 
 editsStored, editsStored.xml


 Implementation of truncate in HDFS-3107 does not allow truncating files which 
 are in a snapshot. It is desirable to be able to truncate and still keep the 
 old file state of the file in the snapshot.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-7831) Fix the starting index of the loop in FileDiffList.findEarlierSnapshotBlocks().

2015-02-24 Thread Konstantin Shvachko (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-7831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Shvachko updated HDFS-7831:
--
Assignee: Konstantin Shvachko
  Status: Patch Available  (was: Open)

 Fix the starting index of the loop in 
 FileDiffList.findEarlierSnapshotBlocks().
 ---

 Key: HDFS-7831
 URL: https://issues.apache.org/jira/browse/HDFS-7831
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Konstantin Shvachko
Assignee: Konstantin Shvachko
 Attachments: HDFS-7831-01.patch


 Currently the loop in {{FileDiffList.findEarlierSnapshotBlocks()}} starts 
 from {{insertPoint + 1}}. It should start from {{insertPoint - 1}}. As noted 
 in [Jing's 
 comment|https://issues.apache.org/jira/browse/HDFS-7056?focusedCommentId=14333864page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14333864]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7308) DFSClient write packet size may 64kB

2015-02-24 Thread Tsz Wo Nicholas Sze (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-7308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14334569#comment-14334569
 ] 

Tsz Wo Nicholas Sze commented on HDFS-7308:
---

Patch looks good to me.

[~stack], I wonder if you could repeat the test you have done for HDFS-7276 
with the patch here to see if the packet size can go over 65536?

 DFSClient write packet size may  64kB
 --

 Key: HDFS-7308
 URL: https://issues.apache.org/jira/browse/HDFS-7308
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs-client
Reporter: Tsz Wo Nicholas Sze
Assignee: Takuya Fukudome
Priority: Minor
 Attachments: HDFS-7308.1.patch


 In DFSOutputStream.computePacketChunkSize(..),
 {code}
   private void computePacketChunkSize(int psize, int csize) {
 final int chunkSize = csize + getChecksumSize();
 chunksPerPacket = Math.max(psize/chunkSize, 1);
 packetSize = chunkSize*chunksPerPacket;
 if (DFSClient.LOG.isDebugEnabled()) {
   ...
 }
   }
 {code}
 We have the following
 || variables || usual values ||
 | psize | dfsClient.getConf().writePacketSize = 64kB |
 | csize | bytesPerChecksum = 512B |
 | getChecksumSize(), i.e. CRC size | 32B |
 | chunkSize = csize + getChecksumSize() | 544B (not a power of two) |
 | psize/chunkSize | 120.47 |
 | chunksPerPacket = max(psize/chunkSize, 1) | 120 |
 | packetSize = chunkSize*chunksPerPacket (not including header) | 65280B |
 | PacketHeader.PKT_MAX_HEADER_LEN | 33B |
 | actual packet size | 65280 + 33 = *65313*  65536 = 64k |
 It is fortunate that the usual packet size = 65313  64k although the 
 calculation above does not guarantee it always happens (e.g. if 
 PKT_MAX_HEADER_LEN=257, then actual packet size=65537  64k.)  We should fix 
 the computation in order to guarantee actual packet size  64k.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-7831) Fix the starting index of the loop in FileDiffList.findEarlierSnapshotBlocks().

2015-02-24 Thread Konstantin Shvachko (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-7831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Shvachko updated HDFS-7831:
--
Attachment: HDFS-7831-01.patch

Fixed the starting index for the loop. Also we do not need to check that {{i  
diffs.size()}}, because now it always is.
This should be treated as an optimization, so there are no additional test 
cases.

 Fix the starting index of the loop in 
 FileDiffList.findEarlierSnapshotBlocks().
 ---

 Key: HDFS-7831
 URL: https://issues.apache.org/jira/browse/HDFS-7831
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Konstantin Shvachko
 Attachments: HDFS-7831-01.patch


 Currently the loop in {{FileDiffList.findEarlierSnapshotBlocks()}} starts 
 from {{insertPoint + 1}}. It should start from {{insertPoint - 1}}. As noted 
 in [Jing's 
 comment|https://issues.apache.org/jira/browse/HDFS-7056?focusedCommentId=14333864page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14333864]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-7832) Show 'Last Modified' in Namenode's 'Browse Filesystem'

2015-02-24 Thread Vinayakumar B (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinayakumar B updated HDFS-7832:

Attachment: HDFS-7832-001.patch

Attaching the changes
Please review

 Show 'Last Modified' in Namenode's 'Browse Filesystem'
 --

 Key: HDFS-7832
 URL: https://issues.apache.org/jira/browse/HDFS-7832
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: namenode
Reporter: Vinayakumar B
Assignee: Vinayakumar B
 Attachments: HDFS-7832-001.patch


 new UI no longer shows the last modified time for a path while browsing.
 This could be added to make browse file system more useful.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

1 2 >

1 - 100 of 120 matches

Mail list logo