date:20140513


 [ 
https://issues.apache.org/jira/browse/HDFS-6345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang updated HDFS-6345:
--

Target Version/s: 2.5.0
  Status: Patch Available  (was: In Progress)

 DFS.listCacheDirectives() should allow filtering based on cache directive ID
 

 Key: HDFS-6345
 URL: https://issues.apache.org/jira/browse/HDFS-6345
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: caching
Affects Versions: 2.4.0
Reporter: Lenni Kuff
Assignee: Andrew Wang
 Attachments: hdfs-6345-1.patch


 DFS.listCacheDirectives() should allow filtering based on cache directive ID. 
 Currently it throws an exception.
 For example:
 {code}
 long directiveId = some id value;
 CacheDirectiveInfo filter = new CacheDirectiveInfo.Builder()  
 .setId(directiveId)
 .build();
 RemoteIteratorCacheDirectiveEntry itr = dfs.listCacheDirectives(filter);
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6372) Handle setXattr rpcIDs for OfflineEditsViewer

Uma Maheswara Rao G created HDFS-6372:
-

 Summary: Handle setXattr rpcIDs for OfflineEditsViewer
 Key: HDFS-6372
 URL: https://issues.apache.org/jira/browse/HDFS-6372
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Affects Versions: HDFS XAttrs (HDFS-2006)
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G


toXml and fromXml methods in SetXattrOp should store and read the rpcids.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6305) WebHdfs response decoding may throw RuntimeExceptions


[ 
https://issues.apache.org/jira/browse/HDFS-6305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996160#comment-13996160
 ] 

Hadoop QA commented on HDFS-6305:
-

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12642531/HDFS-6305.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-hdfs-project/hadoop-hdfs.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HDFS-Build/6889//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6889//console

This message is automatically generated.

 WebHdfs response decoding may throw RuntimeExceptions
 -

 Key: HDFS-6305
 URL: https://issues.apache.org/jira/browse/HDFS-6305
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Priority: Critical
 Attachments: HDFS-6305.patch


 WebHdfs does not guard against exceptions while decoding the response 
 payload.  The json parser will throw RunTime exceptions on malformed 
 responses.  The json decoding routines do not validate the expected fields 
 are present which may cause NPEs.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6381) Fix a typo in INodeReference.java

2014-05-13 Thread Binglin Chang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-6381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Binglin Chang updated HDFS-6381:


Status: Patch Available  (was: Open)

 Fix a typo in INodeReference.java
 -

 Key: HDFS-6381
 URL: https://issues.apache.org/jira/browse/HDFS-6381
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Binglin Chang
Assignee: Binglin Chang
Priority: Trivial
 Attachments: HDFS-6381.v1.patch


 hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java
 {code}
   * For example,
 - * (1) Support we have /abc/foo, say the inode of foo is 
 inode(id=1000,name=foo)
 + * (1) Suppose we have /abc/foo, say the inode of foo is 
 inode(id=1000,name=foo)
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6325) Append should fail if the last block has insufficient number of replicas


[ 
https://issues.apache.org/jira/browse/HDFS-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996013#comment-13996013
 ] 

Hadoop QA commented on HDFS-6325:
-

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12644527/HDFS-6325.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-hdfs-project/hadoop-hdfs.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HDFS-Build/6885//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6885//console

This message is automatically generated.

 Append should fail if the last block has insufficient number of replicas
 

 Key: HDFS-6325
 URL: https://issues.apache.org/jira/browse/HDFS-6325
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 2.2.0
Reporter: Konstantin Shvachko
Assignee: Keith Pak
 Attachments: HDFS-6325.patch, HDFS-6325.patch, HDFS-6325.patch, 
 HDFS-6325_test.patch, appendTest.patch


 Currently append() succeeds on a file with the last block that has no 
 replicas. But the subsequent updatePipeline() fails as there are no replicas 
 with the exception Unable to retrieve blocks locations for last block. This 
 leaves the file unclosed, and others can not do anything with it until its 
 lease expires.
 The solution is to check replicas of the last block on the NameNode and fail 
 during append() rather than during updatePipeline().
 How many replicas should be present before NN allows to append? I see two 
 options:
 # min-replication: allow append if the last block is minimally replicated (1 
 by default)
 # full-replication: allow append if the last block is fully replicated (3 by 
 default)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6328) Simplify code in FSDirectory

2014-05-13 Thread Jing Zhao (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-6328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13993786#comment-13993786
 ] 

Jing Zhao commented on HDFS-6328:
-

The latest patch looks good to me. Only one nit: the comment // src[i - 1] is 
the last common ancestor. should be moved after the while loop. Other than 
this +1 (Maybe you do not need to run Jenkins again after moving the comment).

 Simplify code in FSDirectory
 

 Key: HDFS-6328
 URL: https://issues.apache.org/jira/browse/HDFS-6328
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Reporter: Haohui Mai
Assignee: Haohui Mai
 Attachments: HDFS-6328.000.patch, HDFS-6328.001.patch, 
 HDFS-6328.002.patch, HDFS-6328.003.patch, HDFS-6328.004.patch


 This jira proposes:
 # Cleaning up dead code in FSDirectory.
 # Simplify the control flows that IntelliJ flags as warnings.
 # Move functions related to resolving paths into one place.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Comment Edited] (HDFS-6377) Unify xattr name and value limits into a single limit

[
https://issues.apache.org/jira/browse/HDFS-6377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995977#comment-13995977
]

Uma Maheswara Rao G edited comment on HDFS-6377 at 5/13/14 2:50 AM:

Thanks Andrew.
Unifying xattr name and value limits into a single limit is OK for me.
I think in existing filesystems it should depend on the implementation, for
example in XFS they are separate limits:
http://en.wikipedia.org/wiki/Extended_file_attributes.

Furthermore, can we modify the configuration name according to [~cnauroth]'s
comments:
{quote}The defaults chosen in this patch look reasonable to me with
consideration of the use case driving xattrs. However, I wonder if the property
name should include fs-limits for consistency with the other metadata
limitation properties: dfs.namenode.fs-limits.max-component-length and
dfs.namenode.fs-limits.max-directory-items. What do you think?{quote}

+1 for me, and I'd like we modify the configuration name.

was (Author: hitliuyi):
Thanks Andrew.
I think unifing xattr name and value limits into a single limit is OK for me.
I think in existing filesystems it should depend on the implementation, for
example in XFS they are separate limits:
http://en.wikipedia.org/wiki/Extended_file_attributes.

+1 for me, and I'd like we modify the configuration name.

Unify xattr name and value limits into a single limit
-

Key: HDFS-6377
URL: https://issues.apache.org/jira/browse/HDFS-6377
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: HDFS XAttrs (HDFS-2006)
Reporter: Andrew Wang
Assignee: Andrew Wang
Attachments: hdfs-6377-1.patch

Instead of having separate limits and config options for the size of an
xattr's name and value, let's use a single limit.

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6367) EnumSetParam$Domain#parse fails for parameter containing more than one enum.


 [ 
https://issues.apache.org/jira/browse/HDFS-6367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Uma Maheswara Rao G updated HDFS-6367:
--

   Resolution: Fixed
Fix Version/s: 2.5.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

I have just committed this to trunk, beanch-2. Thanks Yi!

 EnumSetParam$Domain#parse fails for parameter containing more than one enum.
 

 Key: HDFS-6367
 URL: https://issues.apache.org/jira/browse/HDFS-6367
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Affects Versions: 3.0.0
Reporter: Yi Liu
Assignee: Yi Liu
 Fix For: 3.0.0, 2.5.0

 Attachments: HDFS-6367.patch


 Fails because additional , 
 
 {noformat}
 java.lang.IllegalArgumentException: No enum const class 
 org.apache.hadoop.fs.Options$Rename.,OVERWRITE
   at java.lang.Enum.valueOf(Enum.java:196)
   at 
 org.apache.hadoop.hdfs.web.resources.EnumSetParam$Domain.parse(EnumSetParam.java:85)
   at 
 org.apache.hadoop.hdfs.web.resources.RenameOptionSetParam.init(RenameOptionSetParam.java:45)
   at 
 org.apache.hadoop.hdfs.web.resources.TestParam.testRenameOptionSetParam(TestParam.java:355)
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6381) Fix a typo in INodeReference.java

2014-05-13 Thread Binglin Chang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-6381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Binglin Chang updated HDFS-6381:


Attachment: HDFS-6381.v1.patch

 Fix a typo in INodeReference.java
 -

 Key: HDFS-6381
 URL: https://issues.apache.org/jira/browse/HDFS-6381
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Binglin Chang
Assignee: Binglin Chang
Priority: Trivial
 Attachments: HDFS-6381.v1.patch


 hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java
 {code}
   * For example,
 - * (1) Support we have /abc/foo, say the inode of foo is 
 inode(id=1000,name=foo)
 + * (1) Suppose we have /abc/foo, say the inode of foo is 
 inode(id=1000,name=foo)
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6344) Maximum limit on the size of an xattr


[ 
https://issues.apache.org/jira/browse/HDFS-6344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995832#comment-13995832
 ] 

Chris Nauroth commented on HDFS-6344:
-

Sorry I'm coming in late after the patch is already committed.

The defaults chosen in this patch look reasonable to me with consideration of 
the use case driving xattrs.  However, I wonder if the property name should 
include fs-limits for consistency with the other metadata limitation 
properties: {{dfs.namenode.fs-limits.max-component-length}} and 
{{dfs.namenode.fs-limits.max-directory-items}}.  What do you think?

 Maximum limit on the size of an xattr
 -

 Key: HDFS-6344
 URL: https://issues.apache.org/jira/browse/HDFS-6344
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Affects Versions: HDFS XAttrs (HDFS-2006)
Reporter: Andrew Wang
Assignee: Yi Liu
 Fix For: HDFS XAttrs (HDFS-2006)

 Attachments: HDFS-6344.patch


 We should support limits on the maximum size of an xattr name/value.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6378) NFS: when portmap/rpcbind is not available, NFS registration should timeout instead of hanging


[ 
https://issues.apache.org/jira/browse/HDFS-6378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995835#comment-13995835
 ] 

Brandon Li commented on HDFS-6378:
--

Ctrl-C can’t kill NFS if portmap is not started:
{noformat}
14/05/08 15:58:06 INFO nfs3.IdUserGroup: Updated user map size:85
14/05/08 15:58:06 INFO nfs3.IdUserGroup: Updated group map size:110
14/05/08 15:58:06 INFO nfs3.WriteManager: Stream timeout is 1ms.
14/05/08 15:58:06 INFO nfs3.WriteManager: Reset stream timeout to minimum value 
1ms.
14/05/08 15:58:06 INFO nfs3.WriteManager: Maximum open streams is 2
14/05/08 15:58:06 INFO nfs3.OpenFileCtxCache: Maximum open streams is 2
2014-05-08 15:58:07.119 java[19176:1003] Unable to load realm info from 
SCDynamicStore
14/05/08 15:58:07 WARN util.NativeCodeLoader: Unable to load native-hadoop 
library for your platform... using builtin-java classes where applicable
14/05/08 15:58:07 INFO nfs3.RpcProgramNfs3: Create new dump directory 
/tmp/.hdfs-nfs
14/05/08 15:58:07 INFO nfs3.Nfs3Base: NFS server port set to: 2049
14/05/08 15:58:07 INFO oncrpc.SimpleUdpServer: Started listening to UDP 
requests at port 9009 for Rpc program: mountd at localhost:9009 with 
workerCount 1
14/05/08 15:58:07 INFO oncrpc.SimpleTcpServer: Started listening to TCP 
requests at port 9009 for Rpc program: mountd at localhost:9009 with 
workerCount 1
^C14/05/08 15:58:57 ERROR nfs3.Nfs3Base: RECEIVED SIGNAL 2: SIGINT
^C14/05/08 15:59:00 ERROR nfs3.Nfs3Base: RECEIVED SIGNAL 2: SIGINT
^C14/05/08 15:59:03 ERROR nfs3.Nfs3Base: RECEIVED SIGNAL 2: SIGINT
{noformat}

The stack trace:
{noformat}
main prio=5 tid=7fe77c000800 nid=0x102a8e000 runnable [102a8d000]
   java.lang.Thread.State: RUNNABLE
at java.net.PlainDatagramSocketImpl.receive0(Native Method)
- locked 77df4def8 (a java.net.PlainDatagramSocketImpl)
at 
java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:188)
- locked 77df4def8 (a java.net.PlainDatagramSocketImpl)
at java.net.DatagramSocket.receive(DatagramSocket.java:725)
- locked 77df4e978 (a java.net.DatagramPacket)
- locked 77df4cfb8 (a java.net.DatagramSocket)
at org.apache.hadoop.oncrpc.SimpleUdpClient.run(SimpleUdpClient.java:65)
at org.apache.hadoop.oncrpc.RpcProgram.register(RpcProgram.java:119)
at org.apache.hadoop.oncrpc.RpcProgram.register(RpcProgram.java:90)
at org.apache.hadoop.mount.MountdBase.start(MountdBase.java:77)
at 
org.apache.hadoop.hdfs.nfs.nfs3.Nfs3.startServiceInternal(Nfs3.java:58)
at org.apache.hadoop.hdfs.nfs.nfs3.Nfs3.startService(Nfs3.java:66)
at org.apache.hadoop.hdfs.nfs.nfs3.Nfs3.main(Nfs3.java:70)
{noformat}

 NFS: when portmap/rpcbind is not available, NFS registration should timeout 
 instead of hanging 
 ---

 Key: HDFS-6378
 URL: https://issues.apache.org/jira/browse/HDFS-6378
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: nfs
Reporter: Brandon Li





--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6377) Unify xattr name and value limits into a single limit


[ 
https://issues.apache.org/jira/browse/HDFS-6377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995977#comment-13995977
 ] 

Yi Liu commented on HDFS-6377:
--

Thanks Andrew. 
I think unifing xattr name and value limits into a single limit is OK for me.  
I think in existing filesystems it should depend on the implementation, for 
example in XFS they are separate limits: 
http://en.wikipedia.org/wiki/Extended_file_attributes. 

Furthermore, can we modify the configuration name according to [~cnauroth]'s 
comments:
{quote}The defaults chosen in this patch look reasonable to me with 
consideration of the use case driving xattrs. However, I wonder if the property 
name should include fs-limits for consistency with the other metadata 
limitation properties: dfs.namenode.fs-limits.max-component-length and 
dfs.namenode.fs-limits.max-directory-items. What do you think?{quote}

+1 for me, and I'd like we modify the configuration name.

 Unify xattr name and value limits into a single limit
 -

 Key: HDFS-6377
 URL: https://issues.apache.org/jira/browse/HDFS-6377
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Affects Versions: HDFS XAttrs (HDFS-2006)
Reporter: Andrew Wang
Assignee: Andrew Wang
 Attachments: hdfs-6377-1.patch


 Instead of having separate limits and config options for the size of an 
 xattr's name and value, let's use a single limit.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Assigned] (HDFS-6377) Unify xattr name and value limits into a single limit


 [ 
https://issues.apache.org/jira/browse/HDFS-6377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang reassigned HDFS-6377:
-

Assignee: Andrew Wang

 Unify xattr name and value limits into a single limit
 -

 Key: HDFS-6377
 URL: https://issues.apache.org/jira/browse/HDFS-6377
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Affects Versions: HDFS XAttrs (HDFS-2006)
Reporter: Andrew Wang
Assignee: Andrew Wang

 Instead of having separate limits and config options for the size of an 
 xattr's name and value, let's use a single limit.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6382) HDFS File/Directory TTL

2014-05-13 Thread Zesheng Wu (JIRA)

[
https://issues.apache.org/jira/browse/HDFS-6382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Zesheng Wu updated HDFS-6382:
-

Description:
In production environment, we always have scenario like this, we want to backup
files on hdfs for some time and then hope to delete these files automatically.
For example, we keep only 1 day's logs on local disk due to limited disk space,
but we need to keep about 1 month's logs in order to debug program bugs, so we
keep all the logs on hdfs and delete logs which are older than 1 month. This is
a typical scenario of HDFS TTL. So here we propose that hdfs can support TTL.

Following are some details of this proposal:
1. HDFS can support TTL on a specified file or directory
2. If a TTL is set on a file, the file will be deleted automatically after the
TTL is expired
3. If a TTL is set on a directory, the child files and directories will be
deleted automatically after the TTL is expired
4. The child file/directory's TTL configuration should override its parent
directory's
5. A global configuration is needed to configure that whether the deleted
files/directories should go to the trash or not
6. A global configuration is needed to configure that whether a directory with
TTL should be deleted when it is emptied by TTL mechanism or not.

was:
In production environment, we always have scenario like this, we want to backup
files on hdfs for some time and then hope to delete theses files automatically.
For example, we keep only 1 day's logs on local disk due to limited disk space,
but we need to keep about 1 month's logs in order to debug program bugs, so we
keep all the logs on hdfs and delete logs which are older than 1 month. This is
a typical scenario of HDFS TTL. So here we propose that hdfs can support TTL.

HDFS File/Directory TTL
---

Key: HDFS-6382
URL: https://issues.apache.org/jira/browse/HDFS-6382
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs-client, namenode
Affects Versions: 2.4.0
Reporter: Zesheng Wu

In production environment, we always have scenario like this, we want to
backup files on hdfs for some time and then hope to delete these files
automatically. For example, we keep only 1 day's logs on local disk due to
limited disk space, but we need to keep about 1 month's logs in order to
debug program bugs, so we keep all the logs on hdfs and delete logs which are
older than 1 month. This is a typical scenario of HDFS TTL. So here we
propose that hdfs can support TTL.
Following are some details of this proposal:
1. HDFS can support TTL on a specified file or directory
2. If a TTL is set on a file, the file will be deleted automatically after
the TTL is expired
3. If a TTL is set on a directory, the child files and directories will be
deleted automatically after the TTL is expired
4. The child file/directory's TTL configuration should override its parent
directory's
5. A global configuration is needed to configure that whether the deleted
files/directories should go to the trash or not
6. A global configuration is needed to configure that whether a directory
with TTL should be deleted when it is emptied by TTL mechanism or not.

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6374) setXAttr should require the user to be the owner of the file or directory


[ 
https://issues.apache.org/jira/browse/HDFS-6374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995988#comment-13995988
 ] 

Yi Liu commented on HDFS-6374:
--

setXAttr/removeXAttr are only for owner, and getXAttr are *not* only for Owner 
and by r permission.

 setXAttr should require the user to be the owner of the file or directory
 -

 Key: HDFS-6374
 URL: https://issues.apache.org/jira/browse/HDFS-6374
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Affects Versions: HDFS XAttrs (HDFS-2006)
Reporter: Andrew Wang
Assignee: Charles Lamb

 From the attr(5) manpage:
 {noformat}
For  this reason, extended user attributes are only allowed for regular
files and directories,  and  access  to  extended  user  attributes  is
restricted  to the owner and to users with appropriate capabilities for
directories with the sticky bit set (see the chmod(1) manual  page  for
an explanation of Sticky Directories).
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6134) Transparent data at rest encryption

2014-05-13 Thread Steve Loughran (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996245#comment-13996245
 ] 

Steve Loughran commented on HDFS-6134:
--

Some side comments
# The ASF don't want to be distributing cryptography libraries; the design has 
to address that distribution model.
# HADOOP-9565 proposes a {{BlobStore}} extending FileSystem to let users know 
that the FS doesn't have the semantics of a real filesystem -for blobstore 
security we'd need the same marker, so apps using a wrapped blobstore will know 
that some operations are non-atomic and potentially O(files)

 Transparent data at rest encryption
 ---

 Key: HDFS-6134
 URL: https://issues.apache.org/jira/browse/HDFS-6134
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: security
Affects Versions: 2.3.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HDFSDataAtRestEncryption.pdf


 Because of privacy and security regulations, for many industries, sensitive 
 data at rest must be in encrypted form. For example: the healthcare industry 
 (HIPAA regulations), the card payment industry (PCI DSS regulations) or the 
 US government (FISMA regulations).
 This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can 
 be used transparently by any application accessing HDFS via Hadoop Filesystem 
 Java API, Hadoop libhdfs C library, or WebHDFS REST API.
 The resulting implementation should be able to be used in compliance with 
 different regulation requirements.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6345) DFS.listCacheDirectives() should allow filtering based on cache directive ID


 [ 
https://issues.apache.org/jira/browse/HDFS-6345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang updated HDFS-6345:
--

Attachment: hdfs-6345-1.patch

Patch attached. I did manual testing with a branch-2 NN and the new client, 
verified that it still filters correctly, and that we get the same exception 
thrown on error.

 DFS.listCacheDirectives() should allow filtering based on cache directive ID
 

 Key: HDFS-6345
 URL: https://issues.apache.org/jira/browse/HDFS-6345
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: caching
Affects Versions: 2.4.0
Reporter: Lenni Kuff
Assignee: Andrew Wang
 Attachments: hdfs-6345-1.patch


 DFS.listCacheDirectives() should allow filtering based on cache directive ID. 
 Currently it throws an exception.
 For example:
 {code}
 long directiveId = some id value;
 CacheDirectiveInfo filter = new CacheDirectiveInfo.Builder()  
 .setId(directiveId)
 .build();
 RemoteIteratorCacheDirectiveEntry itr = dfs.listCacheDirectives(filter);
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6379) HTTPFS - Implement ACLs support

2014-05-13 Thread Alejandro Abdelnur (JIRA)

Alejandro Abdelnur created HDFS-6379:


 Summary: HTTPFS - Implement ACLs support
 Key: HDFS-6379
 URL: https://issues.apache.org/jira/browse/HDFS-6379
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Fix For: 2.4.0


HDFS-4685 added ACLs support to WebHDFS but missed adding them to HttpFS.

This JIRA is for such.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6305) WebHdfs response decoding may throw RuntimeExceptions

2014-05-13 Thread Jonathan Eagles (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-6305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995737#comment-13995737
 ] 

Jonathan Eagles commented on HDFS-6305:
---

Kicked jenkins. there are 3 or 4 builds ahead of this one. It may be late 
tonight to see the results.

Jon

 WebHdfs response decoding may throw RuntimeExceptions
 -

 Key: HDFS-6305
 URL: https://issues.apache.org/jira/browse/HDFS-6305
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Priority: Critical
 Attachments: HDFS-6305.patch


 WebHdfs does not guard against exceptions while decoding the response 
 payload.  The json parser will throw RunTime exceptions on malformed 
 responses.  The json decoding routines do not validate the expected fields 
 are present which may cause NPEs.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6230) Expose upgrade status through NameNode web UI

2014-05-13 Thread Haohui Mai (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-6230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995340#comment-13995340
 ] 

Haohui Mai commented on HDFS-6230:
--

+1. I'll commit it shortly.

 Expose upgrade status through NameNode web UI
 -

 Key: HDFS-6230
 URL: https://issues.apache.org/jira/browse/HDFS-6230
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 2.4.0
Reporter: Arpit Agarwal
Assignee: Mit Desai
 Attachments: HDFS-6230-NoUpgradesInProgress.png, 
 HDFS-6230-UpgradeInProgress.jpg, HDFS-6230.patch, HDFS-6230.patch


 The NameNode web UI does not show upgrade information anymore. Hadoop 2.0 
 also does not have the _hadoop dfsadmin -upgradeProgress_ command to check 
 the upgrade status.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6382) HDFS File/Directory TTL

2014-05-13 Thread Zesheng Wu (JIRA)

Zesheng Wu created HDFS-6382:


 Summary: HDFS File/Directory TTL
 Key: HDFS-6382
 URL: https://issues.apache.org/jira/browse/HDFS-6382
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs-client, namenode
Affects Versions: 2.4.0
Reporter: Zesheng Wu


In production environment, we always have scenario like this, we want to backup 
files on hdfs for some time and then hope to delete theses files automatically. 
For example, we keep only 1 day's logs on local disk due to limited disk space, 
but we need to keep about 1 month's logs in order to debug program bugs, so we 
keep all the logs on hdfs and delete logs which are older than 1 month. This is 
a typical scenario of HDFS TTL. So here we propose that hdfs can support TTL.

Following are some details of this proposal:
1. HDFS can support TTL on a specified file or directory
2. If a TTL is set on a file, the file will be deleted automatically after the 
TTL is expired
3. If a TTL is set on a directory, the child files and directories will be 
deleted automatically after the TTL is expired
4. The child file/directory's TTL configuration should override its parent 
directory's
5. A global configuration is needed to configure that whether the deleted 
files/directories should go to the trash or not
6. A global configuration is needed to configure that whether a directory with 
TTL should be deleted when it is emptied by TTL mechanism or not.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6328) Clean up dead code in FSDirectory

2014-05-13 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-6328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995409#comment-13995409
 ] 

Hudson commented on HDFS-6328:
--

SUCCESS: Integrated in Hadoop-trunk-Commit #5603 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/5603/])
HDFS-6328. Clean up dead code in FSDirectory. Contributed by Haohui Mai. 
(wheat9: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1593755)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java


 Clean up dead code in FSDirectory
 -

 Key: HDFS-6328
 URL: https://issues.apache.org/jira/browse/HDFS-6328
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Reporter: Haohui Mai
Assignee: Haohui Mai
 Attachments: HDFS-6328.000.patch, HDFS-6328.001.patch, 
 HDFS-6328.002.patch, HDFS-6328.003.patch, HDFS-6328.004.patch


 This jira proposes:
 # Cleaning up dead code in FSDirectory.
 # Simplify the control flows that IntelliJ flags as warnings.
 # Move functions related to resolving paths into one place.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6293) Issues with OIV processing PB-based fsimages

[
https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995444#comment-13995444
]

Kihwal Lee commented on HDFS-6293:
--

bq. can you please mark the class that are brought back deprecated?
I am not sure deprecating it now is a good idea. There are two apsects in this.
* Ability to read the old fsimage format in OIV. This has always been the case
for OIV. Now it will be the combination of oiv and oiv_legacy that will
continue this. As long as NN can read/upgrade old images, there is no reason
to hurry and remove this capability from oiv.
* Support for the use case built around the old oiv. Users won't care much
about the image format as long as they can obtain what they want, using
reasonable and comparable amount of resources as before. If we only think
about this, we can deprecate OIV and NN's ability to save the old format, once
we have a long term alternative solution.

Issues with OIV processing PB-based fsimages

Key: HDFS-6293
URL: https://issues.apache.org/jira/browse/HDFS-6293
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Kihwal Lee
Assignee: Haohui Mai
Priority: Blocker
Attachments: HDFS-6293.000.patch, HDFS-6293.001.patch,
HDFS-6293.002-save-deprecated-fsimage.patch,
HDFS-6293_sbn_ckpt_retention.patch,
HDFS-6293_sbn_ckpt_retention_oiv_legacy.patch, Heap Histogram.html

There are issues with OIV when processing fsimages in protobuf.
Due to the internal layout changes introduced by the protobuf-based fsimage,
OIV consumes excessive amount of memory. We have tested with a fsimage with
about 140M files/directories. The peak heap usage when processing this image
in pre-protobuf (i.e. pre-2.4.0) format was about 350MB. After converting
the image to the protobuf format on 2.4.0, OIV would OOM even with 80GB of
heap (max new size was 1GB). It should be possible to process any image with
the default heap size of 1.5GB.
Another issue is the complete change of format/content in OIV's XML output.
I also noticed that the secret manager section has no tokens while there were
unexpired tokens in the original image (pre-2.4.0). I did not check whether
they were also missing in the new pb fsimage.

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6325) Append should fail if the last block has insufficient number of replicas


[ 
https://issues.apache.org/jira/browse/HDFS-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995837#comment-13995837
 ] 

Hadoop QA commented on HDFS-6325:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12644461/HDFS-6325.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-hdfs-project/hadoop-hdfs:

  
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
  org.apache.hadoop.hdfs.TestFileAppend4

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HDFS-Build/6879//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6879//console

This message is automatically generated.

 Append should fail if the last block has insufficient number of replicas
 

 Key: HDFS-6325
 URL: https://issues.apache.org/jira/browse/HDFS-6325
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 2.2.0
Reporter: Konstantin Shvachko
Assignee: Keith Pak
 Attachments: HDFS-6325.patch, HDFS-6325.patch, HDFS-6325_test.patch, 
 appendTest.patch


 Currently append() succeeds on a file with the last block that has no 
 replicas. But the subsequent updatePipeline() fails as there are no replicas 
 with the exception Unable to retrieve blocks locations for last block. This 
 leaves the file unclosed, and others can not do anything with it until its 
 lease expires.
 The solution is to check replicas of the last block on the NameNode and fail 
 during append() rather than during updatePipeline().
 How many replicas should be present before NN allows to append? I see two 
 options:
 # min-replication: allow append if the last block is minimally replicated (1 
 by default)
 # full-replication: allow append if the last block is fully replicated (3 by 
 default)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6377) Unify xattr name and value limits into a single limit


[ 
https://issues.apache.org/jira/browse/HDFS-6377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996396#comment-13996396
 ] 

Charles Lamb commented on HDFS-6377:


In general, LGTM. One nit (piling on Uma's comment):

+  XAttr is too big, maximum size =  + nnConf.xattrMaxSize
+  + , but the size is =  + xAttr.getName().length());

Could we change the wording to

The XAttr is too big. The maximum size of the name + value is 
nnConf.xattrMaxSize, but the total size is xAttr.getName().length().

+1 from me modulo that.


 Unify xattr name and value limits into a single limit
 -

 Key: HDFS-6377
 URL: https://issues.apache.org/jira/browse/HDFS-6377
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Affects Versions: HDFS XAttrs (HDFS-2006)
Reporter: Andrew Wang
Assignee: Andrew Wang
 Attachments: hdfs-6377-1.patch


 Instead of having separate limits and config options for the size of an 
 xattr's name and value, let's use a single limit.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6345) DFS.listCacheDirectives() should allow filtering based on cache directive ID


[ 
https://issues.apache.org/jira/browse/HDFS-6345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996149#comment-13996149
 ] 

Hadoop QA commented on HDFS-6345:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12644544/hdfs-6345-1.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-hdfs-project/hadoop-hdfs:

  org.apache.hadoop.hdfs.TestDistributedFileSystem

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HDFS-Build/6888//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6888//console

This message is automatically generated.

 DFS.listCacheDirectives() should allow filtering based on cache directive ID
 

 Key: HDFS-6345
 URL: https://issues.apache.org/jira/browse/HDFS-6345
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: caching
Affects Versions: 2.4.0
Reporter: Lenni Kuff
Assignee: Andrew Wang
 Attachments: hdfs-6345-1.patch


 DFS.listCacheDirectives() should allow filtering based on cache directive ID. 
 Currently it throws an exception.
 For example:
 {code}
 long directiveId = some id value;
 CacheDirectiveInfo filter = new CacheDirectiveInfo.Builder()  
 .setId(directiveId)
 .build();
 RemoteIteratorCacheDirectiveEntry itr = dfs.listCacheDirectives(filter);
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6186) Pause deletion of blocks when the namenode starts up


[ 
https://issues.apache.org/jira/browse/HDFS-6186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996194#comment-13996194
 ] 

Hadoop QA commented on HDFS-6186:
-

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12644511/HDFS-6186.003.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-hdfs-project/hadoop-hdfs.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HDFS-Build/6890//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6890//console

This message is automatically generated.

 Pause deletion of blocks when the namenode starts up
 

 Key: HDFS-6186
 URL: https://issues.apache.org/jira/browse/HDFS-6186
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Reporter: Suresh Srinivas
Assignee: Jing Zhao
 Attachments: HDFS-6186.000.patch, HDFS-6186.002.patch, 
 HDFS-6186.003.patch


 HDFS namenode can delete blocks very quickly, given the deletion happens as a 
 parallel operation spread across many datanodes. One of the frequent 
 anxieties I see is that a lot of data can be deleted very quickly, when a 
 cluster is brought up, especially when one of the storage directories has 
 failed and namenode metadata was copied from another storage. Copying wrong 
 metadata would results in some of the newer files (if old metadata was 
 copied) being deleted along with their blocks. 
 HDFS-5986 now captures the number of pending deletion block on namenode webUI 
 and JMX. I propose pausing deletion of blocks for a configured period of time 
 (default 1 hour?) after namenode comes out of safemode. This will give enough 
 time for the administrator to notice large number of pending deletion blocks 
 and take corrective action.
 Thoughts?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6373) Remove support for extended attributes on symlinks


[ 
https://issues.apache.org/jira/browse/HDFS-6373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996425#comment-13996425
 ] 

Charles Lamb commented on HDFS-6373:


While investigating this, I noticed this in the TestNameNodeXAttr output. I 
suspect it's not intended, but I'll investigate further:

2014-05-13 10:04:33,584 INFO  hdfs.StateChange 
(FSNamesystem.java:completeFile(3028)) - DIR* completeFile: /symdir2/target is 
closed by DFSClient_NONMAPREDUCE_239063403_1
2014-05-13 10:04:33,590 INFO  FSNamesystem.audit 
(FSNamesystem.java:logAuditMessage(7980)) - allowed=true   ugi=cwl 
(auth:SIMPLE)   ip=/127.0.0.1   cmd=createSymlink   src=/symdir1/link   
dst=/symdir2/target perm=cwl:supergroup:rwxrwxrwx   proto=rpc
2014-05-13 10:04:33,601 INFO  FSNamesystem.audit 
(FSNamesystem.java:logAuditMessage(7980)) - allowed=true   ugi=cwl 
(auth:SIMPLE)   ip=/127.0.0.1   cmd=setXAttrsrc=/symdir2/target 
dst=nullperm=cwl:supergroup:rw-r--r--   proto=rpc
2014-05-13 10:04:33,605 INFO  FSNamesystem.audit 
(FSNamesystem.java:logAuditMessage(7980)) - allowed=true   ugi=cwl 
(auth:SIMPLE)   ip=/127.0.0.1   cmd=setXAttrsrc=/symdir2/target 
dst=nullperm=cwl:supergroup:rw-r--r--   proto=rpc
2014-05-13 10:04:33,608 INFO  ipc.Server (Server.java:run(2120)) - IPC Server 
handler 9 on 38622, call 
org.apache.hadoop.hdfs.protocol.ClientProtocol.getXAttrs from 127.0.0.1:41722 
Call#18 Retry#0
org.apache.hadoop.hdfs.protocol.UnresolvedPathException: /symdir2/target
at 
org.apache.hadoop.hdfs.server.namenode.INodesInPath.resolve(INodesInPath.java:203)
at 
org.apache.hadoop.hdfs.server.namenode.FSDirectory.getLastINodeInPath(FSDirectory.java:3135)
at 
org.apache.hadoop.hdfs.server.namenode.FSDirectory.getXAttrs(FSDirectory.java:2952)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getXAttrs(FSNamesystem.java:7858)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getXAttrs(NameNodeRpcServer.java:1396)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getXAttrs(ClientNamenodeProtocolServerSideTranslatorPB.java:1294)
at 
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:605)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:932)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2099)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2095)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1608)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2093)

 Remove support for extended attributes on symlinks
 --

 Key: HDFS-6373
 URL: https://issues.apache.org/jira/browse/HDFS-6373
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Reporter: Andrew Wang
Assignee: Charles Lamb

 Looking in the Linux source code, we see the following:
 http://lxr.linux.no/linux+v3.14.3/fs/xattr.c
 {code}
   60/*
   61 * In the user.* namespace, only regular files and directories 
 can have
   62 * extended attributes. For sticky directories, only the owner and
   63 * privileged users can write attributes.
   64 */
 {code}
 We should consider removing {{XAttrFeature}} from {{INodeSymlink}}.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6383) Upgrade S3n s3.fs.buffer.dir to suppoer multi directories

2014-05-13 Thread Ted Malaska (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-6383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Malaska updated HDFS-6383:
--

Status: Patch Available  (was: Open)

Patch is ready

 Upgrade S3n s3.fs.buffer.dir to suppoer multi directories
 -

 Key: HDFS-6383
 URL: https://issues.apache.org/jira/browse/HDFS-6383
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Ted Malaska
Priority: Minor
 Attachments: HDFS-6383.patch


 s3.fs.buffer.dir defines the tmp folder where files will be written to before 
 getting sent to S3.  Right now this is limited to a single folder which 
 causes to major issues.
 1. You need a drive with enough space to store all the tmp files at once
 2. You are limited to the IO speeds of a single drive
 This solution will resolve both and has been tested to increase the S3 write 
 speed by 2.5x with 10 mappers on hs1.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6293) Issues with OIV processing PB-based fsimages

[
https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996442#comment-13996442
]

Kihwal Lee commented on HDFS-6293:
--

Here is recap.
* oiv_legacy for debugging and inspecting old fsimages. As long as NN is
capable of reading and upgrading from it, we should keep it.
* NN's ability to save in old fsimage format: This should go away once we have
a more sustainable solution. Postprocessing ([~wheat9] 's proposal) will cover
many use cases. We may need something else. This is to be determined.
* Keep adding features to the old fsimage format: Please don't!
* Changing the layout of the new PB-based fsimage back to the old way: Heck no!

We will deprecate NN's ability to save in the old fsimage format, once we agree
on what the long term solution is. We should create a separate JIRA and
continue discussion.

As [~sureshms] suggested, I will prepare the patch for trunk as well with
[~ajisakaa]'s comment incorporated.

Issues with OIV processing PB-based fsimages

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6381) Fix a typo in INodeReference.java

2014-05-13 Thread Junping Du (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-6381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996213#comment-13996213
 ] 

Junping Du commented on HDFS-6381:
--

Kick off Jenkins test manually.

 Fix a typo in INodeReference.java
 -

 Key: HDFS-6381
 URL: https://issues.apache.org/jira/browse/HDFS-6381
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Binglin Chang
Assignee: Binglin Chang
Priority: Trivial
 Attachments: HDFS-6381.v1.patch


 hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java
 {code}
   * For example,
 - * (1) Support we have /abc/foo, say the inode of foo is 
 inode(id=1000,name=foo)
 + * (1) Suppose we have /abc/foo, say the inode of foo is 
 inode(id=1000,name=foo)
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6371) In HA setup, the standby NN should redirect WebHDFS write requests to the active NN

2014-05-13 Thread Daryn Sharp (JIRA)

[
https://issues.apache.org/jira/browse/HDFS-6371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996394#comment-13996394
]

Daryn Sharp commented on HDFS-6371:
---

Agreed on client side retries. With IP-based failover, a standby NN that
attempts to redirect to the other NN will create a redirect loop to itself
until the IP failover to the active occurs.

In HA setup, the standby NN should redirect WebHDFS write requests to the
active NN
---

Key: HDFS-6371
URL: https://issues.apache.org/jira/browse/HDFS-6371
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode, webhdfs
Reporter: Tsz Wo Nicholas Sze
Assignee: Tsz Wo Nicholas Sze

The current WebHDFS implementation in namenode does not check its HA state --
it does the same thing no matter it is active or standby.
Suppose a http client talk to the standby NN via WebHDFS. For the read
operations, there is no problem. For the write operations, if the operation
requires http redirect (e.g. creating a file), it will work since the standby
NN will also redirect the client to a DN. When the client connect to the DN,
the DN will fulfill the request with the active NN. However, for the write
operations not requiring http redirect (e.g. mkdir), the operation will fail
with StandbyException since it will be executed with the standby NN.
There are two solutions:
# The http client could catch StandbyException and then retries with the
other NN in this case.
# The standby NN redirects the request to the active NN.
The second solution seems better since the client does not need to know both
active NN and standby NN.
Note that WebHdfsFileSystem is already able to handle HA failover. The JIRA
is for other http clients.

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Assigned] (HDFS-6293) Issues with OIV processing PB-based fsimages

[
https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Kihwal Lee reassigned HDFS-6293:

Assignee: Kihwal Lee (was: Haohui Mai)

Issues with OIV processing PB-based fsimages

Key: HDFS-6293
URL: https://issues.apache.org/jira/browse/HDFS-6293
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Priority: Blocker
Attachments: HDFS-6293.000.patch, HDFS-6293.001.patch,
HDFS-6293.002-save-deprecated-fsimage.patch,
HDFS-6293_sbn_ckpt_retention.patch,
HDFS-6293_sbn_ckpt_retention_oiv_legacy.patch, Heap Histogram.html

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6326) WebHdfs ACL compatibility is broken


[ 
https://issues.apache.org/jira/browse/HDFS-6326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995821#comment-13995821
 ] 

Chris Nauroth commented on HDFS-6326:
-

I've repeated my compatibility testing on patch v4 and confirmed that it still 
works.

 WebHdfs ACL compatibility is broken
 ---

 Key: HDFS-6326
 URL: https://issues.apache.org/jira/browse/HDFS-6326
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Affects Versions: 3.0.0, 2.4.0
Reporter: Daryn Sharp
Assignee: Chris Nauroth
Priority: Blocker
 Attachments: HDFS-6326.1.patch, HDFS-6326.2.patch, HDFS-6326.3.patch, 
 HDFS-6326.4.patch


 2.4 ACL support is completely incompatible with 2.4 webhdfs servers.  The NN 
 throws an {{IllegalArgumentException}} exception.
 {code}
 hadoop fs -ls webhdfs://nn/
 Found 21 items
 ls: Invalid value for webhdfs parameter op: No enum constant 
 org.apache.hadoop.hdfs.web.resources.GetOpParam.Op.GETACLSTATUS
 [... 20 more times...]
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6250) TestBalancerWithNodeGroup.testBalancerWithRackLocality fails

2014-05-13 Thread Junping Du (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-6250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996207#comment-13996207
 ] 

Junping Du commented on HDFS-6250:
--

Hi [~decster], patch LGTM. Would you run 50 iterations and update us your 
results here? Thx!

 TestBalancerWithNodeGroup.testBalancerWithRackLocality fails
 

 Key: HDFS-6250
 URL: https://issues.apache.org/jira/browse/HDFS-6250
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Kihwal Lee
Assignee: Chen He
 Attachments: HDFS-6250-v2.patch, HDFS-6250-v3.patch, HDFS-6250.patch, 
 test_log.txt


 It was seen in https://builds.apache.org/job/PreCommit-HDFS-Build/6669/
 {panel}
 java.lang.AssertionError: expected:1800 but was:1810
   at org.junit.Assert.fail(Assert.java:93)
   at org.junit.Assert.failNotEquals(Assert.java:647)
   at org.junit.Assert.assertEquals(Assert.java:128)
   at org.junit.Assert.assertEquals(Assert.java:147)
   at org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
  .testBalancerWithRackLocality(TestBalancerWithNodeGroup.java:253)
 {panel}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-5682) Heterogeneous Storage phase 2 - APIs to expose Storage Types

2014-05-13 Thread Zesheng Wu (JIRA)

[
https://issues.apache.org/jira/browse/HDFS-5682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996072#comment-13996072
]

Zesheng Wu commented on HDFS-5682:
--

Thanks [~ndimiduk], here are some details of our SLA requirements:
The background is that we have a structured data storage system which is very
similar with amazon DynamoDB and is built on HBase. This system is intended to
serve multi-tenants, in order to ensure the SLA, such as 99% read/write
latency, we need to place the data on SSD. One straightforward way is to set up
a hdfs cluster with all SSD data disks, but this is cost too much. So we need
to reduce the cost. Another way is to set up a hdfs cluster with both SSD and
HDD data disks, we can put 1 replica on SSD, and the other 2 on HDD. This is
really a typical scenario of hdfs heterogeneous storage.

Heterogeneous Storage phase 2 - APIs to expose Storage Types

Key: HDFS-5682
URL: https://issues.apache.org/jira/browse/HDFS-5682
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Arpit Agarwal
Assignee: Arpit Agarwal

Phase 1 (HDFS-2832) added support to present the DataNode as a collection of
discrete storages of different types.
This Jira is to track phase 2 of the Heterogeneous Storage work which
involves exposing Storage Types to applications and adding Quota Management
support for administrators.
This phase will also include tools support for administrators/users.

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Comment Edited] (HDFS-6293) Issues with OIV processing PB-based fsimages


[ 
https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996649#comment-13996649
 ] 

Kihwal Lee edited comment on HDFS-6293 at 5/13/14 5:19 PM:
---

Oops. The patch applies to trunk but it introduces a duplicate import in the 
test case. I will upload separate patches.


was (Author: kihwal):
Ouch. The patch applies to trunk but it introduces a duplicate import in the 
test case. I will upload separate patches.

 Issues with OIV processing PB-based fsimages
 

 Key: HDFS-6293
 URL: https://issues.apache.org/jira/browse/HDFS-6293
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Priority: Blocker
 Attachments: HDFS-6293.000.patch, HDFS-6293.001.patch, 
 HDFS-6293.002-save-deprecated-fsimage.patch, HDFS-6293.branch-2.patch, 
 HDFS-6293.trunk.patch, HDFS-6293.trunk.patch, 
 HDFS-6293_sbn_ckpt_retention.patch, 
 HDFS-6293_sbn_ckpt_retention_oiv_legacy.patch, Heap Histogram.html


 There are issues with OIV when processing fsimages in protobuf. 
 Due to the internal layout changes introduced by the protobuf-based fsimage, 
 OIV consumes excessive amount of memory.  We have tested with a fsimage with 
 about 140M files/directories. The peak heap usage when processing this image 
 in pre-protobuf (i.e. pre-2.4.0) format was about 350MB.  After converting 
 the image to the protobuf format on 2.4.0, OIV would OOM even with 80GB of 
 heap (max new size was 1GB).  It should be possible to process any image with 
 the default heap size of 1.5GB.
 Another issue is the complete change of format/content in OIV's XML output.  
 I also noticed that the secret manager section has no tokens while there were 
 unexpired tokens in the original image (pre-2.4.0).  I did not check whether 
 they were also missing in the new pb fsimage.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6357) SetXattr should persist rpcIDs for handling retrycache with Namenode restart and HA


[ 
https://issues.apache.org/jira/browse/HDFS-6357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13994293#comment-13994293
 ] 

Uma Maheswara Rao G commented on HDFS-6357:
---

Thanks a lot, for the reviews Yi and Andrew!
I have just committed this to branch!

 SetXattr should persist rpcIDs for handling retrycache with Namenode restart 
 and HA
 ---

 Key: HDFS-6357
 URL: https://issues.apache.org/jira/browse/HDFS-6357
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Affects Versions: HDFS XAttrs (HDFS-2006)
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
 Fix For: HDFS XAttrs (HDFS-2006)

 Attachments: HDFS-6357.patch, HDFS-6357.patch, HDFS-6357.patch


 SetXattr should persist rpcIDs for handling restart Namenode and HA scenarios



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6340) DN can't finalize upgarde

2014-05-13 Thread Arpit Agarwal (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-6340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13992964#comment-13992964
 ] 

Arpit Agarwal commented on HDFS-6340:
-

It looks like the previous build aborted. I kicked off a new one.

 DN can't finalize upgarde
 -

 Key: HDFS-6340
 URL: https://issues.apache.org/jira/browse/HDFS-6340
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: datanode
Affects Versions: 2.4.0
Reporter: Rahul Singhal
Priority: Blocker
 Fix For: 3.0.0, 2.4.1

 Attachments: HDFS-6340-branch-2.4.0.patch, HDFS-6340.02.patch, 
 HDFS-6340.patch


 I upgraded a (NN) HA cluster from 2.2.0 to 2.4.0. After I issued the 
 '-finalizeUpgarde' command, NN was able to finalize the upgrade but DN 
 couldn't (I waited for the next block report).
 I think I have found the problem to be due to HDFS-5153. I will attach a 
 proposed fix.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6361) TestIdUserGroup.testUserUpdateSetting failed due to out of range nfsnobody Id

2014-05-13 Thread Yongjun Zhang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-6361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yongjun Zhang updated HDFS-6361:


Attachment: HDFS-6361.003.patch

 TestIdUserGroup.testUserUpdateSetting failed due to out of range nfsnobody Id
 -

 Key: HDFS-6361
 URL: https://issues.apache.org/jira/browse/HDFS-6361
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: nfs
Affects Versions: 2.4.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
 Attachments: HDFS-6361.001.patch, HDFS-6361.002.patch, 
 HDFS-6361.003.patch


 The following error happens pretty often:
 org.apache.hadoop.nfs.nfs3.TestIdUserGroup.testUserUpdateSetting
 Failing for the past 1 build (Since Unstable#61 )
 Took 0.1 sec.
 add description
 Error Message
 For input string: 4294967294
 Stacktrace
 java.lang.NumberFormatException: For input string: 4294967294
   at 
 java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
   at java.lang.Integer.parseInt(Integer.java:495)
   at java.lang.Integer.valueOf(Integer.java:582)
   at 
 org.apache.hadoop.nfs.nfs3.IdUserGroup.updateMapInternal(IdUserGroup.java:137)
   at 
 org.apache.hadoop.nfs.nfs3.IdUserGroup.updateMaps(IdUserGroup.java:188)
   at org.apache.hadoop.nfs.nfs3.IdUserGroup.init(IdUserGroup.java:60)
   at 
 org.apache.hadoop.nfs.nfs3.TestIdUserGroup.testUserUpdateSetting(TestIdUserGroup.java:71)
 Standard Output
 log4j:WARN No appenders could be found for logger 
 (org.apache.hadoop.nfs.nfs3.IdUserGroup).
 log4j:WARN Please initialize the log4j system properly.
 log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
 info.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6056) Clean up NFS config settings


[ 
https://issues.apache.org/jira/browse/HDFS-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996698#comment-13996698
 ] 

Brandon Li commented on HDFS-6056:
--

[~jingzhao], thanks for the review. I've update the jira description and marked 
it as incompatible change.
I've uploaded a new patch to address your comments.

 Clean up NFS config settings
 

 Key: HDFS-6056
 URL: https://issues.apache.org/jira/browse/HDFS-6056
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: nfs
Affects Versions: 2.3.0
Reporter: Aaron T. Myers
Assignee: Brandon Li
 Attachments: HDFS-6056.001.patch, HDFS-6056.002.patch, 
 HDFS-6056.003.patch, HDFS-6056.004.patch


 As discussed on HDFS-6050, there's a few opportunities to improve the config 
 settings related to NFS. This JIRA is to implement those changes, which 
 include: moving hdfs-nfs related properties into hadoop-hdfs-nfs project, and 
 replacing 'nfs' with 'nfs' in the property names.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6370) Web UI fails to display in intranet under IE


 [ 
https://issues.apache.org/jira/browse/HDFS-6370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Nauroth updated HDFS-6370:


  Component/s: namenode
   journal-node
   datanode
 Target Version/s: 3.0.0, 2.5.0
Affects Version/s: 3.0.0
   2.4.0
 Hadoop Flags: Reviewed

 Web UI fails to display in intranet under IE
 

 Key: HDFS-6370
 URL: https://issues.apache.org/jira/browse/HDFS-6370
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: datanode, journal-node, namenode
Affects Versions: 3.0.0, 2.4.0
Reporter: Haohui Mai
Assignee: Haohui Mai
 Attachments: HDFS-6370.000.patch


 When IE renders the web UI of a cluster than runs in the intranet, it forces 
 the compatibility mode to be turned on which causes the UI fails to render 
 correctly.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6367) DomainE extends EnumE#parse in EnumSetParam fails for parmater containing more than one enum.


 [ 
https://issues.apache.org/jira/browse/HDFS-6367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yi Liu updated HDFS-6367:
-

Attachment: HDFS-6367.patch

The , should not be contained.

 DomainE extends EnumE#parse in EnumSetParam fails for parmater containing 
 more than one enum.
 -

 Key: HDFS-6367
 URL: https://issues.apache.org/jira/browse/HDFS-6367
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Affects Versions: 3.0.0
Reporter: Yi Liu
 Fix For: 3.0.0

 Attachments: HDFS-6367.patch


 Fails because additional , 
 
 java.lang.IllegalArgumentException: No enum const class 
 org.apache.hadoop.fs.Options$Rename.,OVERWRITE
   at java.lang.Enum.valueOf(Enum.java:196)
   at 
 org.apache.hadoop.hdfs.web.resources.EnumSetParam$Domain.parse(EnumSetParam.java:85)
   at 
 org.apache.hadoop.hdfs.web.resources.RenameOptionSetParam.init(RenameOptionSetParam.java:45)
   at 
 org.apache.hadoop.hdfs.web.resources.TestParam.testRenameOptionSetParam(TestParam.java:355)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
   at 
 org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
   at 
 org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
   at 
 org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
   at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
   at 
 org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
   at 
 org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6326) WebHdfs ACL compatibility is broken


 [ 
https://issues.apache.org/jira/browse/HDFS-6326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Nauroth updated HDFS-6326:


Attachment: HDFS-6326.5.patch

Daryn, that looks like a good suggestion to me.  Here is patch v5 incorporating 
those ideas.  Mostly, this is just folding your code right into the v4 patch 
with some additional comments.  Note however that I kept the 
{{FsAclPermission}} class as {{Private}} inside the HDFS package, not Common.  
I'd prefer not to publish this into the public API footprint.

I was hesitant to add a boolean member to track true/false for ACL presence 
because of the risk of increasing memory footprint.  However, I confirmed with 
jmap -histo that it makes no difference after your suggested change:

{code}
375: 1 32  org.apache.hadoop.fs.permission.FsPermission
382: 1 32  
org.apache.hadoop.hdfs.protocol.FsAclPermission
{code}

Memory utilization of {{FsPermission}} and {{FsAclPermission}} are very much 
dominated by Java object overhead/padding, so adding the extra boolean didn't 
make a difference.  Haohui and I have had some offline discussion about 
optimizations here, but that's not directly relevant to this issue.

[~daryn], can you please let me know if you're still +1 for v5?  Thanks!

 WebHdfs ACL compatibility is broken
 ---

 Key: HDFS-6326
 URL: https://issues.apache.org/jira/browse/HDFS-6326
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Affects Versions: 3.0.0, 2.4.0
Reporter: Daryn Sharp
Assignee: Chris Nauroth
Priority: Blocker
 Attachments: HDFS-6326.1.patch, HDFS-6326.2.patch, HDFS-6326.3.patch, 
 HDFS-6326.4.patch, HDFS-6326.5.patch, aclfsperm.example


 2.4 ACL support is completely incompatible with 2.4 webhdfs servers.  The NN 
 throws an {{IllegalArgumentException}} exception.
 {code}
 hadoop fs -ls webhdfs://nn/
 Found 21 items
 ls: Invalid value for webhdfs parameter op: No enum constant 
 org.apache.hadoop.hdfs.web.resources.GetOpParam.Op.GETACLSTATUS
 [... 20 more times...]
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6230) Expose upgrade status through NameNode web UI

2014-05-13 Thread Mit Desai (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-6230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mit Desai updated HDFS-6230:


Attachment: HDFS-6230.patch

Thanks for looking at the patch [~wheat9]. Posting the updated patch.

 Expose upgrade status through NameNode web UI
 -

 Key: HDFS-6230
 URL: https://issues.apache.org/jira/browse/HDFS-6230
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 2.4.0
Reporter: Arpit Agarwal
Assignee: Mit Desai
 Attachments: HDFS-6230-NoUpgradesInProgress.png, 
 HDFS-6230-UpgradeInProgress.jpg, HDFS-6230.patch, HDFS-6230.patch


 The NameNode web UI does not show upgrade information anymore. Hadoop 2.0 
 also does not have the _hadoop dfsadmin -upgradeProgress_ command to check 
 the upgrade status.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6369) RemoteBlockReader#available() should call FSInputChecker.available()

2014-05-13 Thread Colin Patrick McCabe (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-6369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995294#comment-13995294
 ] 

Colin Patrick McCabe commented on HDFS-6369:


The calling code in {{DFSInputStream}} already takes into account where the end 
of the block is.  See this code in {{DFSInputStream}}:
{code}
if (pos = targetPos  targetPos = blockEnd) {
  //
  // If this seek is to a positive position in the current
  // block, and this piece of data might already be lying in
  // the TCP buffer, then just eat up the intervening data.
  //
  int diff = (int)(targetPos - pos);
  if (diff = blockReader.available()) {
  ...
{code}

Notice that we don't call {{available}} if {{targetPos}} is beyond {{blockEnd}}

Also, {{RemoteBlockReader}} is deprecated.  See the comment at the top:

{code}
/**
 * \@deprecated this is an old implementation that is being left around
 * in case any issues spring up with the new {\@link RemoteBlockReader2} 
implementation.
 * It will be removed in the next release.
 */
{code}

We should really fix {{RemoteBlockReader2}} to support SOCKS sockets (those are 
the only ones without channels, as I recall) and remove {{RemoteBlockReader}} 
as the comment promised we'd do.

 RemoteBlockReader#available() should call FSInputChecker.available()
 

 Key: HDFS-6369
 URL: https://issues.apache.org/jira/browse/HDFS-6369
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu
Assignee: Chen He
Priority: Trivial
 Attachments: HDFS-6369.patch


 Currently DFSClient.TCP_WINDOW_SIZE is directly returned.
 However, FSInputChecker.available(), in the superclass, may return value 
 lower than the constant.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6346) Optimize OP_SET_XATTRS by persisting single Xattr entry per setXattr/removeXattr api call

[
https://issues.apache.org/jira/browse/HDFS-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Yi Liu updated HDFS-6346:
-

Attachment: HDFS-6346.2.patch

Uma, thanks for review and collecting the improvement data. That's great. The
new patch includes update for your comment.

Optimize OP_SET_XATTRS by persisting single Xattr entry per
setXattr/removeXattr api call
-

Key: HDFS-6346
URL: https://issues.apache.org/jira/browse/HDFS-6346
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Reporter: Uma Maheswara Rao G
Assignee: Yi Liu
Attachments: HDFS-6346.1.patch, HDFS-6346.2.patch, HDFS-6346.patch,
editsStored

When a new xattrs set on an Inode, it may add this with OP_SET_XATTRS and
let's say [user.name1:value1]
On a next call if we set another xattrs, then it may store along with older
existing xattrs again. It may be like [user.name1:value1, user.name2:value2]
So, on adding more xattrs on same Inode, that list may grow and we keep store
the entries of already existing name, value fairs.
Right now we defaulted the max Xattrs on an Inode to 32 and configured. If
user modified it to much larger value and start setting xattrs, then edits
loading may create issue like my above example.
But I didn't refer any usecase of having large number of xattrs, this is just
the assumption to consider a case. My biggest doubt is whether we will have
such real usecases to have huge xattrs on a single INode.
So, here is a thought on having OP_SET_XATTR for each setXAttr operation to
be logged, When we replay them we need to consolidate. This is some initial
thought we can think more if others also feel we need to consider this case
to handle.
Otherwise we endup storing Xattrs entries in editlog file as n(n+1)/2 where n
is number xattrs for a file/dir. This may be issue only when we have large
number configured max xattrs for inode.

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6186) Pause deletion of blocks when the namenode starts up

2014-05-13 Thread Suresh Srinivas (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-6186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996688#comment-13996688
 ] 

Suresh Srinivas commented on HDFS-6186:
---

+1 for the patch.

We should create another jira to print along with current pending deletion 
count displayed on namenode webUI, something like:
{{Pending Deletion Blocks:  (Block deletion will resume after xxx seconds)}}

 Pause deletion of blocks when the namenode starts up
 

 Key: HDFS-6186
 URL: https://issues.apache.org/jira/browse/HDFS-6186
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Reporter: Suresh Srinivas
Assignee: Jing Zhao
 Attachments: HDFS-6186.000.patch, HDFS-6186.002.patch, 
 HDFS-6186.003.patch


 HDFS namenode can delete blocks very quickly, given the deletion happens as a 
 parallel operation spread across many datanodes. One of the frequent 
 anxieties I see is that a lot of data can be deleted very quickly, when a 
 cluster is brought up, especially when one of the storage directories has 
 failed and namenode metadata was copied from another storage. Copying wrong 
 metadata would results in some of the newer files (if old metadata was 
 copied) being deleted along with their blocks. 
 HDFS-5986 now captures the number of pending deletion block on namenode webUI 
 and JMX. I propose pausing deletion of blocks for a configured period of time 
 (default 1 hour?) after namenode comes out of safemode. This will give enough 
 time for the administrator to notice large number of pending deletion blocks 
 and take corrective action.
 Thoughts?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HDFS-6380) TestBalancerWithNodeGroup is failing in trunk


 [ 
https://issues.apache.org/jira/browse/HDFS-6380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Uma Maheswara Rao G resolved HDFS-6380.
---

Resolution: Duplicate

 TestBalancerWithNodeGroup is failing in trunk
 -

 Key: HDFS-6380
 URL: https://issues.apache.org/jira/browse/HDFS-6380
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 3.0.0
Reporter: Uma Maheswara Rao G

 Error Message
  expected:1800 but was:1814
 Stacktrace
 {noformat}
 java.lang.AssertionError: expected:1800 but was:1814
   at org.junit.Assert.fail(Assert.java:88)
   at org.junit.Assert.failNotEquals(Assert.java:743)
   at org.junit.Assert.assertEquals(Assert.java:118)
   at org.junit.Assert.assertEquals(Assert.java:144)
   at 
 org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup.testBalancerWithRackLocality(TestBalancerWithNodeGroup.java:253)
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6377) Unify xattr name and value limits into a single limit


[ 
https://issues.apache.org/jira/browse/HDFS-6377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995997#comment-13995997
 ] 

Uma Maheswara Rao G commented on HDFS-6377:
---


Thanks a lot, Andrew for filing this JIRA and for the patch.
Adding to Liu's  and Chris comments, I have following comments:
- . {code}
 XAttr is too big, maximum size =  + nnConf.xattrMaxSize
+  + , but the size is =  + xAttr.getName().length());
{code}

Seems like now max size is combination of name and value size together, but 
here you considered on name#length?

- . {code}
 public static void init() throws Exception {
+conf = new Configuration();
{code}
Lets use HdfsCOnfiguration? [ but no harm in using either :-) ]

- . {code}
 fs.setXAttr(path, name1, longValue);
+  fs.setXAttr(path, user.a, longValue);
{code}
you can use name1 for it?

- 
Do you think we need to have minimum configuration limit?  Lets say user 
configured size as 3, then this is always invalid size as Namespace itself 
occupy this space?  [ I am not insisting, just to discuss this point ]
Atleast we can warn on startup saying, you can not add xattrs with configured 
limit? [ I think 7 is the minimum size for adding xattr with user name space 
where name and value can have one char each at least]

- 
Thanks for the other cleanup.
Yes, I agree with Chris, having fs-limits for consistency make sense to me.

 Unify xattr name and value limits into a single limit
 -

 Key: HDFS-6377
 URL: https://issues.apache.org/jira/browse/HDFS-6377
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Affects Versions: HDFS XAttrs (HDFS-2006)
Reporter: Andrew Wang
Assignee: Andrew Wang
 Attachments: hdfs-6377-1.patch


 Instead of having separate limits and config options for the size of an 
 xattr's name and value, let's use a single limit.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6056) Clean up NFS config settings


 [ 
https://issues.apache.org/jira/browse/HDFS-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brandon Li updated HDFS-6056:
-

Description: As discussed on HDFS-6050, there's a few opportunities to 
improve the config settings related to NFS. This JIRA is to implement those 
changes, which include: moving hdfs-nfs related properties into hadoop-hdfs-nfs 
project, and replacing 'nfs' with 'nfs' in the property names.  (was: As 
discussed on HDFS-6050, there's a few opportunities to improve the config 
settings related to NFS. This JIRA is to implement those changes.)

 Clean up NFS config settings
 

 Key: HDFS-6056
 URL: https://issues.apache.org/jira/browse/HDFS-6056
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: nfs
Affects Versions: 2.3.0
Reporter: Aaron T. Myers
Assignee: Brandon Li
 Attachments: HDFS-6056.001.patch, HDFS-6056.002.patch, 
 HDFS-6056.003.patch, HDFS-6056.004.patch


 As discussed on HDFS-6050, there's a few opportunities to improve the config 
 settings related to NFS. This JIRA is to implement those changes, which 
 include: moving hdfs-nfs related properties into hadoop-hdfs-nfs project, and 
 replacing 'nfs' with 'nfs' in the property names.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6355) Fix divide-by-zero, improper use of wall-clock time in BlockPoolSliceScanner

2014-05-13 Thread Colin Patrick McCabe (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-6355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Patrick McCabe updated HDFS-6355:
---

   Resolution: Fixed
Fix Version/s: 2.5.0
   Status: Resolved  (was: Patch Available)

committed, thanks

 Fix divide-by-zero, improper use of wall-clock time in BlockPoolSliceScanner
 

 Key: HDFS-6355
 URL: https://issues.apache.org/jira/browse/HDFS-6355
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
 Fix For: 2.5.0

 Attachments: HDFS-6355.001.patch


 BlockPoolSliceScanner uses {{Time.now}} to calculate an interval.  But this 
 is incorrect, since if the wall-clock time changes, we will end up setting 
 the scan periods to a shorter or longer time than we configured.
 There is also a case where we may divide by zero if we get unlucky, because 
 we calculate an interval and divide by it, without checking whether the 
 interval is 0 milliseconds.  This would produce an {{ArithmeticException}} 
 since we are using longs.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6367) EnumSetParam$Domain#parse fails for parameter containing more than one enum.


 [ 
https://issues.apache.org/jira/browse/HDFS-6367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Uma Maheswara Rao G updated HDFS-6367:
--

Description: 
Fails because additional , 

java.lang.IllegalArgumentException: No enum const class 
org.apache.hadoop.fs.Options$Rename.,OVERWRITE
at java.lang.Enum.valueOf(Enum.java:196)
at 
org.apache.hadoop.hdfs.web.resources.EnumSetParam$Domain.parse(EnumSetParam.java:85)
at 
org.apache.hadoop.hdfs.web.resources.RenameOptionSetParam.init(RenameOptionSetParam.java:45)
at 
org.apache.hadoop.hdfs.web.resources.TestParam.testRenameOptionSetParam(TestParam.java:355)


  was:
Fails because additional , 

java.lang.IllegalArgumentException: No enum const class 
org.apache.hadoop.fs.Options$Rename.,OVERWRITE
at java.lang.Enum.valueOf(Enum.java:196)
at 
org.apache.hadoop.hdfs.web.resources.EnumSetParam$Domain.parse(EnumSetParam.java:85)
at 
org.apache.hadoop.hdfs.web.resources.RenameOptionSetParam.init(RenameOptionSetParam.java:45)
at 
org.apache.hadoop.hdfs.web.resources.TestParam.testRenameOptionSetParam(TestParam.java:355)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at 
org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at 
org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at 
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at 
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at 
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at 
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)




 EnumSetParam$Domain#parse fails for parameter containing more than one enum.
 

 Key: HDFS-6367
 URL: https://issues.apache.org/jira/browse/HDFS-6367
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Affects Versions: 3.0.0
Reporter: Yi Liu
Assignee: Yi Liu
 Fix For: 3.0.0

 Attachments: HDFS-6367.patch


 Fails because additional , 
 
 java.lang.IllegalArgumentException: No enum const class 
 org.apache.hadoop.fs.Options$Rename.,OVERWRITE
   at java.lang.Enum.valueOf(Enum.java:196)
   at 
 org.apache.hadoop.hdfs.web.resources.EnumSetParam$Domain.parse(EnumSetParam.java:85)
   at 
 org.apache.hadoop.hdfs.web.resources.RenameOptionSetParam.init(RenameOptionSetParam.java:45)
   at 
 org.apache.hadoop.hdfs.web.resources.TestParam.testRenameOptionSetParam(TestParam.java:355)
   



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6383) Upgrade S3n s3.fs.buffer.dir to suppoer multi directories

2014-05-13 Thread Ted Malaska (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-6383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Malaska updated HDFS-6383:
--

Attachment: HDFS-6383.patch

This solves the problem.

Thanks to:
Joe Prosser, Dave Wang, Andrei Savu
and 
Govind Kamat for testing

 Upgrade S3n s3.fs.buffer.dir to suppoer multi directories
 -

 Key: HDFS-6383
 URL: https://issues.apache.org/jira/browse/HDFS-6383
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Ted Malaska
Priority: Minor
 Attachments: HDFS-6383.patch


 s3.fs.buffer.dir defines the tmp folder where files will be written to before 
 getting sent to S3.  Right now this is limited to a single folder which 
 causes to major issues.
 1. You need a drive with enough space to store all the tmp files at once
 2. You are limited to the IO speeds of a single drive
 This solution will resolve both and has been tested to increase the S3 write 
 speed by 2.5x with 10 mappers on hs1.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6293) Issues with OIV processing PB-based fsimages

[
https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996649#comment-13996649
]

Kihwal Lee commented on HDFS-6293:
--

Ouch. The patch applies to trunk but it introduces a duplicate import in the
test case. I will upload separate patches.

Issues with OIV processing PB-based fsimages

Key: HDFS-6293
URL: https://issues.apache.org/jira/browse/HDFS-6293
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Priority: Blocker
Attachments: HDFS-6293.000.patch, HDFS-6293.001.patch,
HDFS-6293.002-save-deprecated-fsimage.patch, HDFS-6293.trunk.patch,
HDFS-6293_sbn_ckpt_retention.patch,
HDFS-6293_sbn_ckpt_retention_oiv_legacy.patch, Heap Histogram.html

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6384) Provide an alternative offline file system analysis tool

Kihwal Lee created HDFS-6384:


 Summary: Provide an alternative offline file system analysis tool
 Key: HDFS-6384
 URL: https://issues.apache.org/jira/browse/HDFS-6384
 Project: Hadoop HDFS
  Issue Type: Improvement
Affects Versions: 2.5.0
Reporter: Kihwal Lee
Priority: Critical


Per discussion in HDFS-6293, we need an alternative way of analyzing file 
system offline.  Some use cases can be covered by post-processing the new 
PB-based fsimage without consuming much resources, but others may need a way to 
efficiently generating fully constructed name space.

Once we have a better solution, we should deprecate NN's ability to save its 
partial state in the old fsimage format.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6370) Web UI fails to display in intranet under IE


[ 
https://issues.apache.org/jira/browse/HDFS-6370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996834#comment-13996834
 ] 

Chris Nauroth commented on HDFS-6370:
-

bq. -1 tests included. The patch doesn't appear to include any new or modified 
tests.

This patch changes HTML only.  There are no new automated tests.  It has been 
tested manually by browsing with IE.

I'm going to commit this.

 Web UI fails to display in intranet under IE
 

 Key: HDFS-6370
 URL: https://issues.apache.org/jira/browse/HDFS-6370
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: datanode, journal-node, namenode
Affects Versions: 3.0.0, 2.4.0
Reporter: Haohui Mai
Assignee: Haohui Mai
 Attachments: HDFS-6370.000.patch


 When IE renders the web UI of a cluster than runs in the intranet, it forces 
 the compatibility mode to be turned on which causes the UI fails to render 
 correctly.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6283) Write end user documentation for xattrs.


[ 
https://issues.apache.org/jira/browse/HDFS-6283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996767#comment-13996767
 ] 

Chris Nauroth commented on HDFS-6283:
-

Hi, Andrew.  The new content looks great.  Here are a few comments:

# Would you please add a hyperlink to the new doc in 
hadoop-project/src/site/site.xml?
# Would you please also update FileSystemShell.apt.vm to add getfattr and 
setfattr?
# The Go Back link is broken.  It looks like some existing Go Back links are 
broken too in the published documentation.  Let's fix this one here, and I 
filed HADOOP-10602 to clean up the others.
# Let's mention that setting a value for an xattr is optional.  (Some might 
find it helpful to set a valueless xattr to use as a kind of tag.)

 Write end user documentation for xattrs.
 

 Key: HDFS-6283
 URL: https://issues.apache.org/jira/browse/HDFS-6283
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: documentation
Reporter: Chris Nauroth
Assignee: Andrew Wang
 Attachments: hdfs-6283-1.patch


 Update the File System Shell documentation to cover the new getfattr and 
 setfattr commands.  If warranted, consider adding a separate dedicated page 
 for fuller discussion of the xattrs model and how the feature works in more 
 detail.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6056) Clean up NFS config settings


 [ 
https://issues.apache.org/jira/browse/HDFS-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brandon Li updated HDFS-6056:
-

Attachment: HDFS-6056.004.patch

 Clean up NFS config settings
 

 Key: HDFS-6056
 URL: https://issues.apache.org/jira/browse/HDFS-6056
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: nfs
Affects Versions: 2.3.0
Reporter: Aaron T. Myers
Assignee: Brandon Li
 Attachments: HDFS-6056.001.patch, HDFS-6056.002.patch, 
 HDFS-6056.003.patch, HDFS-6056.004.patch


 As discussed on HDFS-6050, there's a few opportunities to improve the config 
 settings related to NFS. This JIRA is to implement those changes.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6345) DFS.listCacheDirectives() should allow filtering based on cache directive ID

2014-05-13 Thread Colin Patrick McCabe (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-6345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996783#comment-13996783
 ] 

Colin Patrick McCabe commented on HDFS-6345:


{code}
+// Querying for a single ID
+final Long id = filter.getId();
+if (id != null) {
+  if (!directivesById.containsKey(id)) {
+throw new InvalidRequestException(Did not find requested id  + id);
+  }
+  prevId = id - 1;
 }
{code}

This is a bit tricky, so I think we should add a comment to the code explaining 
why we're changing prevId.

{code}
+
 ArrayListCacheDirectiveEntry replies =
 new ArrayListCacheDirectiveEntry(NUM_PRE_ALLOCATED_ENTRIES);
 int numReplies = 0;
@@ -710,6 +718,11 @@ public void removeDirective(long id, FSPermissionChecker 
pc)
   }
   CacheDirective curDirective = cur.getValue();
   CacheDirectiveInfo info = cur.getValue().toInfo();
+  if (id != null 
+  !(info.getId().equals(id))) {
+// If we didn't find the requested ID, we're done
+break;
+  }
{code}

Maybe change the comment to if we didn't find, or filtered out the requested 
ID.  Filtering by ID is only one kind of filtering... if an additional filter 
is set which eliminates the requested item, nothing will show up.

 DFS.listCacheDirectives() should allow filtering based on cache directive ID
 

 Key: HDFS-6345
 URL: https://issues.apache.org/jira/browse/HDFS-6345
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: caching
Affects Versions: 2.4.0
Reporter: Lenni Kuff
Assignee: Andrew Wang
 Attachments: hdfs-6345-1.patch


 DFS.listCacheDirectives() should allow filtering based on cache directive ID. 
 Currently it throws an exception.
 For example:
 {code}
 long directiveId = some id value;
 CacheDirectiveInfo filter = new CacheDirectiveInfo.Builder()  
 .setId(directiveId)
 .build();
 RemoteIteratorCacheDirectiveEntry itr = dfs.listCacheDirectives(filter);
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6373) Remove support for extended attributes on symlinks


[ 
https://issues.apache.org/jira/browse/HDFS-6373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996905#comment-13996905
 ] 

Andrew Wang commented on HDFS-6373:
---

This is basically what I had in mind, but how do you feel about throwing some 
kind of RuntimeException, e.g. UnsupportedOperationException? It should prevent 
anyone from trying this in the future.

 Remove support for extended attributes on symlinks
 --

 Key: HDFS-6373
 URL: https://issues.apache.org/jira/browse/HDFS-6373
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Reporter: Andrew Wang
Assignee: Charles Lamb
 Attachments: HDFS-6373.1.patch


 Looking in the Linux source code, we see the following:
 http://lxr.linux.no/linux+v3.14.3/fs/xattr.c
 {code}
   60/*
   61 * In the user.* namespace, only regular files and directories 
 can have
   62 * extended attributes. For sticky directories, only the owner and
   63 * privileged users can write attributes.
   64 */
 {code}
 We should consider removing {{XAttrFeature}} from {{INodeSymlink}}.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6134) Transparent data at rest encryption

2014-05-13 Thread Alejandro Abdelnur (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995608#comment-13995608
 ] 

Alejandro Abdelnur commented on HDFS-6134:
--

Owen, I think the docs posted in HADOOP-10150 will address most if not all your 
questions/concerns:

https://issues.apache.org/jira/secure/attachment/12640388/HDFSDataAtRestEncryptionAlternatives.pdf

https://issues.apache.org/jira/secure/attachment/12640389/HDFSDataatRestEncryptionProposal.pdf

https://issues.apache.org/jira/secure/attachment/12640390/HDFSDataatRestEncryptionAttackVectors.pdf

 Transparent data at rest encryption
 ---

 Key: HDFS-6134
 URL: https://issues.apache.org/jira/browse/HDFS-6134
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: security
Affects Versions: 2.3.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HDFSDataAtRestEncryption.pdf


 Because of privacy and security regulations, for many industries, sensitive 
 data at rest must be in encrypted form. For example: the healthcare industry 
 (HIPAA regulations), the card payment industry (PCI DSS regulations) or the 
 US government (FISMA regulations).
 This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can 
 be used transparently by any application accessing HDFS via Hadoop Filesystem 
 Java API, Hadoop libhdfs C library, or WebHDFS REST API.
 The resulting implementation should be able to be used in compliance with 
 different regulation requirements.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6378) NFS: when portmap/rpcbind is not available, NFS registration should timeout instead of hanging


 [ 
https://issues.apache.org/jira/browse/HDFS-6378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brandon Li updated HDFS-6378:
-

Description: NFS gateway should  

 NFS: when portmap/rpcbind is not available, NFS registration should timeout 
 instead of hanging 
 ---

 Key: HDFS-6378
 URL: https://issues.apache.org/jira/browse/HDFS-6378
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: nfs
Reporter: Brandon Li

 NFS gateway should  



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6293) Issues with OIV processing PB-based fsimages

2014-05-13 Thread Suresh Srinivas (JIRA)

[
https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996922#comment-13996922
]

Suresh Srinivas commented on HDFS-6293:
---

+1 for the patch. Thanks [~hmai] and [~kihwal] for making these changes.

Issues with OIV processing PB-based fsimages

Key: HDFS-6293
URL: https://issues.apache.org/jira/browse/HDFS-6293
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Priority: Blocker
Attachments: HDFS-6293.000.patch, HDFS-6293.001.patch,
HDFS-6293.002-save-deprecated-fsimage.patch, HDFS-6293.branch-2.patch,
HDFS-6293.trunk.patch, HDFS-6293.trunk.patch, HDFS-6293.v2.branch-2.patch,
HDFS-6293.v2.trunk.patch, HDFS-6293_sbn_ckpt_retention.patch,
HDFS-6293_sbn_ckpt_retention_oiv_legacy.patch, Heap Histogram.html

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6370) Web UI fails to display in intranet under IE


[ 
https://issues.apache.org/jira/browse/HDFS-6370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996809#comment-13996809
 ] 

Hadoop QA commented on HDFS-6370:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12644321/HDFS-6370.000.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-hdfs-project/hadoop-hdfs.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HDFS-Build/6893//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6893//console

This message is automatically generated.

 Web UI fails to display in intranet under IE
 

 Key: HDFS-6370
 URL: https://issues.apache.org/jira/browse/HDFS-6370
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: datanode, journal-node, namenode
Affects Versions: 3.0.0, 2.4.0
Reporter: Haohui Mai
Assignee: Haohui Mai
 Attachments: HDFS-6370.000.patch


 When IE renders the web UI of a cluster than runs in the intranet, it forces 
 the compatibility mode to be turned on which causes the UI fails to render 
 correctly.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6373) Remove support for extended attributes on symlinks


 [ 
https://issues.apache.org/jira/browse/HDFS-6373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Charles Lamb updated HDFS-6373:
---

Attachment: HDFS-6373.2.patch

 Remove support for extended attributes on symlinks
 --

 Key: HDFS-6373
 URL: https://issues.apache.org/jira/browse/HDFS-6373
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Reporter: Andrew Wang
Assignee: Charles Lamb
 Attachments: HDFS-6373.1.patch, HDFS-6373.2.patch


 Looking in the Linux source code, we see the following:
 http://lxr.linux.no/linux+v3.14.3/fs/xattr.c
 {code}
   60/*
   61 * In the user.* namespace, only regular files and directories 
 can have
   62 * extended attributes. For sticky directories, only the owner and
   63 * privileged users can write attributes.
   64 */
 {code}
 We should consider removing {{XAttrFeature}} from {{INodeSymlink}}.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6395) Assorted improvements to xattr limit checking


[ 
https://issues.apache.org/jira/browse/HDFS-6395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996989#comment-13996989
 ] 

Andrew Wang commented on HDFS-6395:
---

Let's also remove the extra import in FSNamesystem that Chris noticed from 
HDFS-6377:

{code}
+import java.io.UnsupportedEncodingException;
{code}

 Assorted improvements to xattr limit checking
 -

 Key: HDFS-6395
 URL: https://issues.apache.org/jira/browse/HDFS-6395
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Affects Versions: HDFS XAttrs (HDFS-2006)
Reporter: Andrew Wang

 It'd be nice to print messages during fsimage and editlog loading if we hit 
 either the # of xattrs per inode or the xattr size limits.
 We should also consider making the # of xattrs limit only apply to the user 
 namespace, or to each namespace separately, to prevent users from locking out 
 access to other namespaces.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Comment Edited] (HDFS-6293) Issues with OIV processing PB-based fsimages

2014-05-13 Thread Suresh Srinivas (JIRA)

[
https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996922#comment-13996922
]

Suresh Srinivas edited comment on HDFS-6293 at 5/13/14 9:36 PM:

+1 for the patch. Thanks [~wheat9] and [~kihwal] for making these changes.

was (Author: sureshms):
+1 for the patch. Thanks [~hmai] and [~kihwal] for making these changes.

Issues with OIV processing PB-based fsimages

Key: HDFS-6293
URL: https://issues.apache.org/jira/browse/HDFS-6293
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Priority: Blocker
Attachments: HDFS-6293.000.patch, HDFS-6293.001.patch,
HDFS-6293.002-save-deprecated-fsimage.patch, HDFS-6293.branch-2.patch,
HDFS-6293.trunk.patch, HDFS-6293.trunk.patch, HDFS-6293.v2.branch-2.patch,
HDFS-6293.v2.trunk.patch, HDFS-6293_sbn_ckpt_retention.patch,
HDFS-6293_sbn_ckpt_retention_oiv_legacy.patch, Heap Histogram.html

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6373) Remove support for extended attributes on symlinks


 [ 
https://issues.apache.org/jira/browse/HDFS-6373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Charles Lamb updated HDFS-6373:
---

Attachment: HDFS-6373.1.patch

Andrew, attached is a patch which overrides the appropriate xattr methods in 
INodeWithAdditionalFields.

 Remove support for extended attributes on symlinks
 --

 Key: HDFS-6373
 URL: https://issues.apache.org/jira/browse/HDFS-6373
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Reporter: Andrew Wang
Assignee: Charles Lamb
 Attachments: HDFS-6373.1.patch


 Looking in the Linux source code, we see the following:
 http://lxr.linux.no/linux+v3.14.3/fs/xattr.c
 {code}
   60/*
   61 * In the user.* namespace, only regular files and directories 
 can have
   62 * extended attributes. For sticky directories, only the owner and
   63 * privileged users can write attributes.
   64 */
 {code}
 We should consider removing {{XAttrFeature}} from {{INodeSymlink}}.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Work started] (HDFS-6346) Optimize OP_SET_XATTRS by persisting single Xattr entry per setXattr/removeXattr api call

[
https://issues.apache.org/jira/browse/HDFS-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Work on HDFS-6346 started by Yi Liu.

Optimize OP_SET_XATTRS by persisting single Xattr entry per
setXattr/removeXattr api call
-

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6370) Web UI fails to display in intranet under IE


 [ 
https://issues.apache.org/jira/browse/HDFS-6370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Nauroth updated HDFS-6370:


   Resolution: Fixed
Fix Version/s: 2.5.0
   3.0.0
   Status: Resolved  (was: Patch Available)

I committed this to trunk and branch-2.  Haohui, thank you for contributing the 
fix.

 Web UI fails to display in intranet under IE
 

 Key: HDFS-6370
 URL: https://issues.apache.org/jira/browse/HDFS-6370
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: datanode, journal-node, namenode
Affects Versions: 3.0.0, 2.4.0
Reporter: Haohui Mai
Assignee: Haohui Mai
 Fix For: 3.0.0, 2.5.0

 Attachments: HDFS-6370.000.patch


 When IE renders the web UI of a cluster than runs in the intranet, it forces 
 the compatibility mode to be turned on which causes the UI fails to render 
 correctly.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6373) Remove support for extended attributes on symlinks


[ 
https://issues.apache.org/jira/browse/HDFS-6373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996912#comment-13996912
 ] 

Charles Lamb commented on HDFS-6373:


Yeah, good idea. I've uploaded a new patch.

 Remove support for extended attributes on symlinks
 --

 Key: HDFS-6373
 URL: https://issues.apache.org/jira/browse/HDFS-6373
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Reporter: Andrew Wang
Assignee: Charles Lamb
 Attachments: HDFS-6373.1.patch, HDFS-6373.2.patch


 Looking in the Linux source code, we see the following:
 http://lxr.linux.no/linux+v3.14.3/fs/xattr.c
 {code}
   60/*
   61 * In the user.* namespace, only regular files and directories 
 can have
   62 * extended attributes. For sticky directories, only the owner and
   63 * privileged users can write attributes.
   64 */
 {code}
 We should consider removing {{XAttrFeature}} from {{INodeSymlink}}.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6313) WebHdfs may use the wrong NN when configured for multiple HA NNs


[ 
https://issues.apache.org/jira/browse/HDFS-6313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13993042#comment-13993042
 ] 

Kihwal Lee commented on HDFS-6313:
--

MiniDFSCluster or NN does not alter the config after its up. It was a easy way 
to get HA config set up for the test.

 WebHdfs may use the wrong NN when configured for multiple HA NNs
 

 Key: HDFS-6313
 URL: https://issues.apache.org/jira/browse/HDFS-6313
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Affects Versions: 3.0.0, 2.4.0
Reporter: Daryn Sharp
Assignee: Kihwal Lee
Priority: Blocker
 Fix For: 3.0.0, 2.4.1

 Attachments: HDFS-6313.branch-2.4.patch, HDFS-6313.patch


 WebHdfs resolveNNAddr will return a union of addresses for all HA configured 
 NNs.  The client may access the wrong NN.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6373) Remove support for extended attributes on symlinks


[ 
https://issues.apache.org/jira/browse/HDFS-6373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996975#comment-13996975
 ] 

Chris Nauroth commented on HDFS-6373:
-

{quote}
This makes sense to me, since symlink permissions are always ignored. Linux 
actually doesn't even let you change symlink permissions, though it is allowed 
by the Unix spec. Chris Nauroth, could you comment on whether this was 
intentional, or just an oversight? It seems like we might want to remove 
ACLs-on-symlinks, which is compatible since symlinks are still hard-disabled.
{quote}

It is impossible to set an ACL on a symlink.  Any ACL operation run on a 
symlink automatically dereferences the symlink and operates on its target, i.e. 
by passing {{true}} to {{FSDirectory#getINodesInPath4Write}}.  We wrote tests 
asserting this behavior, so I don't think there is any way to store an 
{{AclFeature}} on an {{INodeSymlink}}.

I quickly skimmed the patch, and it looks like you're going for some additional 
prevention inside {{INodeSymlink}} to prevent mistakes.  That seems like a good 
idea, and we could do similar for ACLs.

 Remove support for extended attributes on symlinks
 --

 Key: HDFS-6373
 URL: https://issues.apache.org/jira/browse/HDFS-6373
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Reporter: Andrew Wang
Assignee: Charles Lamb
 Attachments: HDFS-6373.1.patch, HDFS-6373.2.patch


 Looking in the Linux source code, we see the following:
 http://lxr.linux.no/linux+v3.14.3/fs/xattr.c
 {code}
   60/*
   61 * In the user.* namespace, only regular files and directories 
 can have
   62 * extended attributes. For sticky directories, only the owner and
   63 * privileged users can write attributes.
   64 */
 {code}
 We should consider removing {{XAttrFeature}} from {{INodeSymlink}}.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6334) Client failover proxy provider for IP failover based NN HA


 [ 
https://issues.apache.org/jira/browse/HDFS-6334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kihwal Lee updated HDFS-6334:
-

   Resolution: Fixed
Fix Version/s: 2.5.0
   3.0.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Thanks for the review, Aaron. I've committed this to trunk and branch-2.

 Client failover proxy provider for IP failover based NN HA
 --

 Key: HDFS-6334
 URL: https://issues.apache.org/jira/browse/HDFS-6334
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Kihwal Lee
Assignee: Kihwal Lee
 Fix For: 3.0.0, 2.5.0

 Attachments: HDFS-6334.patch, HDFS-6334.v2.patch, HDFS-6334.v3.patch


 With RPCv9 and improvements in the SPNEGO auth handling, it is possible to 
 set up a pair of HA namenodes utilizing IP failover as client-request fencing 
 mechanism.
 This jira will make it possible for HA to be configured without requiring use 
 of logical URI and provide a simple IP failover proxy provider.  The change 
 will allow any old implementation of {{FailoverProxyProvider}} to continue to 
 work.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6283) Write end user documentation for xattrs.


 [ 
https://issues.apache.org/jira/browse/HDFS-6283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang updated HDFS-6283:
--

Attachment: hdfs-6283-2.patch

 Write end user documentation for xattrs.
 

 Key: HDFS-6283
 URL: https://issues.apache.org/jira/browse/HDFS-6283
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: documentation
Reporter: Chris Nauroth
Assignee: Andrew Wang
 Attachments: hdfs-6283-1.patch, hdfs-6283-2.patch


 Update the File System Shell documentation to cover the new getfattr and 
 setfattr commands.  If warranted, consider adding a separate dedicated page 
 for fuller discussion of the xattrs model and how the feature works in more 
 detail.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6293) Issues with OIV processing PB-based fsimages

[
https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996582#comment-13996582
]

Hadoop QA commented on HDFS-6293:
-

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12644631/HDFS-6293.trunk.patch
against trunk revision .

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.

{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.

Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6892//console

This message is automatically generated.

Issues with OIV processing PB-based fsimages

Key: HDFS-6293
URL: https://issues.apache.org/jira/browse/HDFS-6293
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Priority: Blocker
Attachments: HDFS-6293.000.patch, HDFS-6293.001.patch,
HDFS-6293.002-save-deprecated-fsimage.patch, HDFS-6293.trunk.patch,
HDFS-6293_sbn_ckpt_retention.patch,
HDFS-6293_sbn_ckpt_retention_oiv_legacy.patch, Heap Histogram.html

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6385) Show when block deletion will start after NameNode startup in WebUI

2014-05-13 Thread Jing Zhao (JIRA)

Jing Zhao created HDFS-6385:
---

 Summary: Show when block deletion will start after NameNode 
startup in WebUI
 Key: HDFS-6385
 URL: https://issues.apache.org/jira/browse/HDFS-6385
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Jing Zhao


HDFS-6186 provides functionality to delay block deletion for a period of time 
after NameNode startup. Currently we only show the number of pending block 
deletions in WebUI. We should also show when the block deletion will start in 
WebUI.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6293) Issues with OIV processing PB-based fsimages


[ 
https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997051#comment-13997051
 ] 

Hadoop QA commented on HDFS-6293:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12644696/HDFS-6293.v2.trunk.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

  {color:red}-1 javac{color}.  The applied patch generated 1277 javac 
compiler warnings (more than the trunk's current 1275 warnings).

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-hdfs-project/hadoop-hdfs:

  org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HDFS-Build/6896//testReport/
Javac warnings: 
https://builds.apache.org/job/PreCommit-HDFS-Build/6896//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6896//console

This message is automatically generated.

 Issues with OIV processing PB-based fsimages
 

 Key: HDFS-6293
 URL: https://issues.apache.org/jira/browse/HDFS-6293
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Priority: Blocker
 Attachments: HDFS-6293.000.patch, HDFS-6293.001.patch, 
 HDFS-6293.002-save-deprecated-fsimage.patch, HDFS-6293.branch-2.patch, 
 HDFS-6293.trunk.patch, HDFS-6293.trunk.patch, HDFS-6293.v2.branch-2.patch, 
 HDFS-6293.v2.trunk.patch, HDFS-6293_sbn_ckpt_retention.patch, 
 HDFS-6293_sbn_ckpt_retention_oiv_legacy.patch, Heap Histogram.html


 There are issues with OIV when processing fsimages in protobuf. 
 Due to the internal layout changes introduced by the protobuf-based fsimage, 
 OIV consumes excessive amount of memory.  We have tested with a fsimage with 
 about 140M files/directories. The peak heap usage when processing this image 
 in pre-protobuf (i.e. pre-2.4.0) format was about 350MB.  After converting 
 the image to the protobuf format on 2.4.0, OIV would OOM even with 80GB of 
 heap (max new size was 1GB).  It should be possible to process any image with 
 the default heap size of 1.5GB.
 Another issue is the complete change of format/content in OIV's XML output.  
 I also noticed that the secret manager section has no tokens while there were 
 unexpired tokens in the original image (pre-2.4.0).  I did not check whether 
 they were also missing in the new pb fsimage.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6293) Issues with OIV processing PB-based fsimages

[
https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Kihwal Lee updated HDFS-6293:
-

Status: Patch Available (was: In Progress)

Issues with OIV processing PB-based fsimages

Key: HDFS-6293
URL: https://issues.apache.org/jira/browse/HDFS-6293
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Priority: Blocker
Attachments: HDFS-6293.000.patch, HDFS-6293.001.patch,
HDFS-6293.002-save-deprecated-fsimage.patch, HDFS-6293.trunk.patch,
HDFS-6293_sbn_ckpt_retention.patch,
HDFS-6293_sbn_ckpt_retention_oiv_legacy.patch, Heap Histogram.html

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6056) Clean up NFS config settings

2014-05-13 Thread Aaron T. Myers (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997056#comment-13997056
 ] 

Aaron T. Myers commented on HDFS-6056:
--

Why not add proper config deprecations of the old config names to the new 
config names so that this won't be an incompatible change? That seems vastly 
preferable to me.

 Clean up NFS config settings
 

 Key: HDFS-6056
 URL: https://issues.apache.org/jira/browse/HDFS-6056
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: nfs
Affects Versions: 2.3.0
Reporter: Aaron T. Myers
Assignee: Brandon Li
 Attachments: HDFS-6056.001.patch, HDFS-6056.002.patch, 
 HDFS-6056.003.patch, HDFS-6056.004.patch


 As discussed on HDFS-6050, there's a few opportunities to improve the config 
 settings related to NFS. This JIRA is to implement those changes, which 
 include: moving hdfs-nfs related properties into hadoop-hdfs-nfs project, and 
 replacing 'nfs' with 'nfs' in the property names.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6377) Unify xattr name and value limits into a single limit


 [ 
https://issues.apache.org/jira/browse/HDFS-6377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang updated HDFS-6377:
--

Attachment: hdfs-6377-2.patch

 Unify xattr name and value limits into a single limit
 -

 Key: HDFS-6377
 URL: https://issues.apache.org/jira/browse/HDFS-6377
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Affects Versions: HDFS XAttrs (HDFS-2006)
Reporter: Andrew Wang
Assignee: Andrew Wang
 Attachments: hdfs-6377-1.patch, hdfs-6377-2.patch


 Instead of having separate limits and config options for the size of an 
 xattr's name and value, let's use a single limit.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6395) Assorted improvements to xattr limit checking

Andrew Wang created HDFS-6395:
-

 Summary: Assorted improvements to xattr limit checking
 Key: HDFS-6395
 URL: https://issues.apache.org/jira/browse/HDFS-6395
 Project: Hadoop HDFS
  Issue Type: Sub-task
Affects Versions: HDFS XAttrs (HDFS-2006)
Reporter: Andrew Wang


It'd be nice to print messages during fsimage and editlog loading if we hit 
either the # of xattrs per inode or the xattr size limits.

We should also consider making the # of xattrs limit only apply to the user 
namespace, or to each namespace separately, to prevent users from locking out 
access to other namespaces.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6283) Write end user documentation for xattrs.


[ 
https://issues.apache.org/jira/browse/HDFS-6283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997053#comment-13997053
 ] 

Andrew Wang commented on HDFS-6283:
---

New patch attached, thanks for the reviews all. I think this addresses all the 
comments except as follows:

- I didn't split out the common changes, seems like we could move this to a 
HADOOP subtask if people prefer, but it seems nicer to do the FsShell updates 
with this same patch as Chris requested. Plus, this way the typo is 
consistently fixed in all three places.
- However, for wording changes, I'd prefer to punt that to another JIRA. I used 
the exactly text from the FsShell help, so we should update all three places 
(FsShell, ExtendedAttributes.apt.vm, FileSystemShell.apt.vm) if we want to 
change the wording anywhere.

 Write end user documentation for xattrs.
 

 Key: HDFS-6283
 URL: https://issues.apache.org/jira/browse/HDFS-6283
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: documentation
Reporter: Chris Nauroth
Assignee: Andrew Wang
 Attachments: hdfs-6283-1.patch, hdfs-6283-2.patch


 Update the File System Shell documentation to cover the new getfattr and 
 setfattr commands.  If warranted, consider adding a separate dedicated page 
 for fuller discussion of the xattrs model and how the feature works in more 
 detail.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6293) Issues with OIV processing PB-based fsimages

[
https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997142#comment-13997142
]

Kihwal Lee commented on HDFS-6293:
--

The failed test is unrelated to this JIRA. It will be fixed in HDFS-6257.
The two warnings (deprecation) are expected.

Issues with OIV processing PB-based fsimages

Key: HDFS-6293
URL: https://issues.apache.org/jira/browse/HDFS-6293
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Priority: Blocker
Attachments: HDFS-6293.000.patch, HDFS-6293.001.patch,
HDFS-6293.002-save-deprecated-fsimage.patch, HDFS-6293.branch-2.patch,
HDFS-6293.trunk.patch, HDFS-6293.trunk.patch, HDFS-6293.v2.branch-2.patch,
HDFS-6293.v2.trunk.patch, HDFS-6293_sbn_ckpt_retention.patch,
HDFS-6293_sbn_ckpt_retention_oiv_legacy.patch, Heap Histogram.html

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.

2014-05-13 Thread samira elhami (JIRA)

[
https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996841#comment-13996841
]

samira elhami commented on HDFS-2576:
-

This patch is really helpful. I know it is added to hadoop-2.1.0-Beta. I've
install this version of hadoop but I don't know how to set the block placement.
I've searched everywhere. please help me.
Thanks

Namenode should have a favored nodes hint to enable clients to have control
over block placement.
-

Key: HDFS-2576
URL: https://issues.apache.org/jira/browse/HDFS-2576
Project: Hadoop HDFS
Issue Type: New Feature
Components: hdfs-client, namenode
Reporter: Pritam Damania
Assignee: Devaraj Das
Fix For: 2.1.0-beta

Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch,
hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch,
hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch,
hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch

Sometimes Clients like HBase are required to dynamically compute the
datanodes it wishes to place the blocks for a file for higher level of
locality. For this purpose there is a need of a way to give the Namenode a
hint in terms of a favoredNodes parameter about the locations where the
client wants to put each block. The proposed solution is a favored nodes
parameter in the addBlock() method and in the create() file method to enable
the clients to give the hints to the NameNode about the locations of each
replica of the block. Note that this would be just a hint and finally the
NameNode would look at disk usage, datanode load etc. and decide whether it
can respect the hints or not.

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6373) Remove support for extended attributes on symlinks


 [ 
https://issues.apache.org/jira/browse/HDFS-6373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang updated HDFS-6373:
--

   Resolution: Fixed
Fix Version/s: HDFS XAttrs (HDFS-2006)
   Status: Resolved  (was: Patch Available)

Committed to branch, thanks all.

 Remove support for extended attributes on symlinks
 --

 Key: HDFS-6373
 URL: https://issues.apache.org/jira/browse/HDFS-6373
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Reporter: Andrew Wang
Assignee: Charles Lamb
 Fix For: HDFS XAttrs (HDFS-2006)

 Attachments: HDFS-6373.1.patch, HDFS-6373.2.patch


 Looking in the Linux source code, we see the following:
 http://lxr.linux.no/linux+v3.14.3/fs/xattr.c
 {code}
   60/*
   61 * In the user.* namespace, only regular files and directories 
 can have
   62 * extended attributes. For sticky directories, only the owner and
   63 * privileged users can write attributes.
   64 */
 {code}
 We should consider removing {{XAttrFeature}} from {{INodeSymlink}}.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6377) Unify xattr name and value limits into a single limit


[ 
https://issues.apache.org/jira/browse/HDFS-6377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997065#comment-13997065
 ] 

Chris Nauroth commented on HDFS-6377:
-

Ah, thanks for the explanation of the {{cacheManager}} bit.  :-)

 Unify xattr name and value limits into a single limit
 -

 Key: HDFS-6377
 URL: https://issues.apache.org/jira/browse/HDFS-6377
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Affects Versions: HDFS XAttrs (HDFS-2006)
Reporter: Andrew Wang
Assignee: Andrew Wang
 Fix For: HDFS XAttrs (HDFS-2006)

 Attachments: hdfs-6377-1.patch, hdfs-6377-2.patch


 Instead of having separate limits and config options for the size of an 
 xattr's name and value, let's use a single limit.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6387) HDFS CLI admin tool for creating deleting an encryption zone

2014-05-13 Thread Alejandro Abdelnur (JIRA)

Alejandro Abdelnur created HDFS-6387:


 Summary: HDFS CLI admin tool for creating  deleting an encryption 
zone
 Key: HDFS-6387
 URL: https://issues.apache.org/jira/browse/HDFS-6387
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode, security
Reporter: Alejandro Abdelnur
Assignee: Charles Lamb


CLI admin tool to create/delete an encryption zone in HDFS.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6293) Issues with OIV processing PB-based fsimages

[
https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996873#comment-13996873
]

Kihwal Lee commented on HDFS-6293:
--

I will fix all warnings except the two coming from FSImage due to the use of
deprecated old image saving methods.

Issues with OIV processing PB-based fsimages

Key: HDFS-6293
URL: https://issues.apache.org/jira/browse/HDFS-6293
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Priority: Blocker
Attachments: HDFS-6293.000.patch, HDFS-6293.001.patch,
HDFS-6293.002-save-deprecated-fsimage.patch, HDFS-6293.branch-2.patch,
HDFS-6293.trunk.patch, HDFS-6293.trunk.patch,
HDFS-6293_sbn_ckpt_retention.patch,
HDFS-6293_sbn_ckpt_retention_oiv_legacy.patch, Heap Histogram.html

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6381) Fix a typo in INodeReference.java

2014-05-13 Thread Chen He (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-6381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chen He updated HDFS-6381:
--

Component/s: documentation

 Fix a typo in INodeReference.java
 -

 Key: HDFS-6381
 URL: https://issues.apache.org/jira/browse/HDFS-6381
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: documentation
Reporter: Binglin Chang
Assignee: Binglin Chang
Priority: Trivial
 Attachments: HDFS-6381.v1.patch


 hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java
 {code}
   * For example,
 - * (1) Support we have /abc/foo, say the inode of foo is 
 inode(id=1000,name=foo)
 + * (1) Suppose we have /abc/foo, say the inode of foo is 
 inode(id=1000,name=foo)
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6376) Distcp data between two HA clusters requires another configuration

2014-05-13 Thread Dave Marion (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-6376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dave Marion updated HDFS-6376:
--

Component/s: hdfs-client
 datanode

 Distcp data between two HA clusters requires another configuration
 --

 Key: HDFS-6376
 URL: https://issues.apache.org/jira/browse/HDFS-6376
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: datanode, federation, hdfs-client
Affects Versions: 2.3.0, 2.4.0
 Environment: CDH 5.0 (Apache 2.3.0 + some patches)
Reporter: Dave Marion
 Fix For: 2.4.1

 Attachments: HDFS-6376-2.patch, HDFS-6376-patch-1.patch


 User has to create a third set of configuration files for distcp when 
 transferring data between two HA clusters.
 Consider the scenario in [1]. You cannot put all of the required properties 
 in core-site.xml and hdfs-site.xml for the client to resolve the location of 
 both active namenodes. If you do, then the datanodes from cluster A may join 
 cluster B. I can not find a configuration option that tells the datanodes to 
 federate blocks for only one of the clusters in the configuration.
 [1] 
 http://mail-archives.apache.org/mod_mbox/hadoop-user/201404.mbox/%3CBAY172-W2133964E0C283968C161DD1520%40phx.gbl%3E



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6375) Listing extended attributes with the search permission


[ 
https://issues.apache.org/jira/browse/HDFS-6375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997120#comment-13997120
 ] 

Charles Lamb commented on HDFS-6375:


Alternatively, we could just look at the permissions and only return the names 
(or names + values) based on the permissions that the caller has.


 Listing extended attributes with the search permission
 --

 Key: HDFS-6375
 URL: https://issues.apache.org/jira/browse/HDFS-6375
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Affects Versions: HDFS XAttrs (HDFS-2006)
Reporter: Andrew Wang
Assignee: Charles Lamb

 From the attr(5) manpage:
 {noformat}
Users with search access to a file or directory may retrieve a list  of
attribute names defined for that file or directory.
 {noformat}
 This is like doing {{getfattr}} without the {{-d}} flag, which we currently 
 don't support.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6396) Remove support for ACL feature from INodeSymlink

Andrew Wang created HDFS-6396:
-

 Summary: Remove support for ACL feature from INodeSymlink
 Key: HDFS-6396
 URL: https://issues.apache.org/jira/browse/HDFS-6396
 Project: Hadoop HDFS
  Issue Type: Improvement
Affects Versions: 2.4.0
Reporter: Andrew Wang
Assignee: Charles Lamb
Priority: Minor


Symlinks cannot have ACLs, but we still have support for the ACL feature in 
INodeSymlink because of class inheritance. Let's remove this support for code 
consistency.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-2006) ability to support storing extended attributes per file

2014-05-13 Thread Tsz Wo Nicholas Sze (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-2006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996507#comment-13996507
 ] 

Tsz Wo Nicholas Sze commented on HDFS-2006:
---

The latest design looks good.

For a XAttr with large value, writing it to editlog may be a problem.  Do we 
plan to have a size limit on the size of XAttr values?

 ability to support storing extended attributes per file
 ---

 Key: HDFS-2006
 URL: https://issues.apache.org/jira/browse/HDFS-2006
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: namenode
Affects Versions: HDFS XAttrs (HDFS-2006)
Reporter: dhruba borthakur
Assignee: Yi Liu
 Attachments: HDFS-XAttrs-Design-1.pdf, HDFS-XAttrs-Design-2.pdf, 
 HDFS-XAttrs-Design-3.pdf, Test-Plan-for-Extended-Attributes-1.pdf, 
 xattrs.1.patch, xattrs.patch


 It would be nice if HDFS provides a feature to store extended attributes for 
 files, similar to the one described here: 
 http://en.wikipedia.org/wiki/Extended_file_attributes. 
 The challenge is that it has to be done in such a way that a site not using 
 this feature does not waste precious memory resources in the namenode.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6383) Upgrade S3n s3.fs.buffer.dir to suppoer multi directories

2014-05-13 Thread David S. Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-6383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997211#comment-13997211
 ] 

David S. Wang commented on HDFS-6383:
-

Thanks Ted for the patch.

We should probably use the LocalDirAllocator like what s3a uses.  That seems to 
be the proper way to do this in Hadoop.

 Upgrade S3n s3.fs.buffer.dir to suppoer multi directories
 -

 Key: HDFS-6383
 URL: https://issues.apache.org/jira/browse/HDFS-6383
 Project: Hadoop HDFS
  Issue Type: Improvement
Affects Versions: 2.4.0
Reporter: Ted Malaska
Assignee: Ted Malaska
Priority: Minor
 Attachments: HDFS-6383.patch


 s3.fs.buffer.dir defines the tmp folder where files will be written to before 
 getting sent to S3.  Right now this is limited to a single folder which 
 causes to major issues.
 1. You need a drive with enough space to store all the tmp files at once
 2. You are limited to the IO speeds of a single drive
 This solution will resolve both and has been tested to increase the S3 write 
 speed by 2.5x with 10 mappers on hs1.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6293) Issues with OIV processing PB-based fsimages