date:20140512


 [ 
https://issues.apache.org/jira/browse/HDFS-6357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Uma Maheswara Rao G updated HDFS-6357:
--

Summary: SetXattr should persist rpcIDs for handling retrycache with 
Namenode restart and HA  (was: SetXattr should persist rpcIDs for handling 
Namenode restart and HA)

 SetXattr should persist rpcIDs for handling retrycache with Namenode restart 
 and HA
 ---

 Key: HDFS-6357
 URL: https://issues.apache.org/jira/browse/HDFS-6357
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Affects Versions: HDFS XAttrs (HDFS-2006)
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
 Attachments: HDFS-6357.patch, HDFS-6357.patch


 SetXattr should persist rpcIDs for handling restart Namenode and HA scenarios



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6294) Use INode IDs to avoid conflicts when a file open for write is renamed


[ 
https://issues.apache.org/jira/browse/HDFS-6294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13992952#comment-13992952
 ] 

Colin Patrick McCabe commented on HDFS-6294:


Looks like I need to check that the inode object is not null before setting the 
path based on it.

 Use INode IDs to avoid conflicts when a file open for write is renamed
 --

 Key: HDFS-6294
 URL: https://issues.apache.org/jira/browse/HDFS-6294
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 0.20.1
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
 Attachments: HDFS-6294.001.patch, HDFS-6294.002.patch, 
 HDFS-6294.003.patch, HDFS-6294.004.patch, HDFS-6294.005.patch


 Now that we have a unique INode ID for each INode, clients with files that 
 are open for write can use this unique ID rather than a file path when they 
 are requesting more blocks or closing the open file.  This will avoid 
 conflicts when a file which is open for write is renamed, and another file 
 with that name is created.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6259) Support extended attributes via WebHDFS/HttpFS


[ 
https://issues.apache.org/jira/browse/HDFS-6259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13994876#comment-13994876
 ] 

Yi Liu commented on HDFS-6259:
--

Require the bug fix in HDFS-6367.

 Support extended attributes via WebHDFS/HttpFS
 --

 Key: HDFS-6259
 URL: https://issues.apache.org/jira/browse/HDFS-6259
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: webhdfs
Affects Versions: HDFS XAttrs (HDFS-2006)
Reporter: Yi Liu
Assignee: Yi Liu
 Attachments: HDFS-6259.1.patch, HDFS-6259.patch


 Get xattrs:
 {code}curl -i 
 'http://HOST:PORT/webhdfs/v1/PATH?op=GETXATTRSencoding=ENCODING'{code}
 Get xattr:
 {code}curl -i 
 'http://HOST:PORT/webhdfs/v1/PATH?op=GETXATTRxattr.name=XATTRNAMEencoding=ENCODING'{code}
 Set xattr:
 {code}curl -i -X PUT 
 'http://HOST:PORT/webhdfs/v1/PATH?op=SETXATTRxattr.name=XATTRNAMExattr.value=XATTRVALUEflag=FLAG'{code}
 Remove xattr:
 {code}curl -i -X PUT 
 'http://HOST:PORT/webhdfs/v1/PATH?op=REMOVEXATTRxattr.name=XATTRNAME'{code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6360) MiniDFSCluster can cause unexpected side effects due to sharing of config

Kihwal Lee created HDFS-6360:


 Summary: MiniDFSCluster can cause unexpected side effects due to 
sharing of config
 Key: HDFS-6360
 URL: https://issues.apache.org/jira/browse/HDFS-6360
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Kihwal Lee


As noted in HDFS-6329 and HDFS-5522, certain use cases of MiniDFSCluster can 
result in unexpected results and falsely failing or passing unit tests.

Since a {{Configuration}} object is shared for all namenode startups, the 
modified conf object during a NN startup is passed to the next NN startup.  The 
effect of the modified conf propagation and subsequent modifications is 
different depending on whether it is a single NN cluster, HA cluster or 
federation cluster.

It also depends on what test cases are doing with the config. For example, 
MiniDFSCluster#getConfiguration(int) returns the saved conf for the specified 
NN, but that is not actually the conf object used by the NN. It just contained 
the same content one time in the past and it is not guaranteed to be that way.

Restarting the same NN can also cause unexpected results. The new NN will 
switch to the conf that was cloned  saved AFTER the last startup.  The new NN 
will start with a changed config intentionally or unintentionally.  The config 
variables such as {{fs.defaultFs}}, {{dfs.namenode.rpc-address}} will be 
implicitly set differently than the initial condition.  Some test cases rely on 
this and others occasionally break because of this.

In summary,
* MiniDFSCluster does not properly isolate configs.
* Many test cases happen to work most of times. Correcting MiniDFSCluster 
causes mass breakages of test cases and requires fixing them.
* Many test cases rely on broken behavior and might pass when they should have 
actually failed.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Reopened] (HDFS-5688) Wire-encription in QJM

2014-05-12 Thread Juan Carlos Fernandez (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-5688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Juan Carlos Fernandez reopened HDFS-5688:
-


Why has being marked as closed? I wasn't able to run QJM + SSL

 Wire-encription in QJM
 --

 Key: HDFS-5688
 URL: https://issues.apache.org/jira/browse/HDFS-5688
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: ha, journal-node, security
Affects Versions: 2.2.0
Reporter: Juan Carlos Fernandez
Priority: Blocker
  Labels: security
 Attachments: core-site.xml, hdfs-site.xml, jaas.conf, journal.xml, 
 namenode.xml, ssl-client.xml, ssl-server.xml


 When HA is implemented with QJM and using kerberos, it's not possible to set 
 wire-encrypted data.
 If it's set property hadoop.rpc.protection to something different to 
 authentication it doesn't work propertly, getting the error:
 ERROR security.UserGroupInformation: PriviledgedActionException 
 as:principal@REALM (auth:KERBEROS) cause:javax.security.sasl.SaslException: 
 No common protection layer between client and server
 With NFS as shared storage everything works like a charm



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6367) EnumSetParam$Domain#parse fails for parameter containing more than one enum.


 [ 
https://issues.apache.org/jira/browse/HDFS-6367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Uma Maheswara Rao G updated HDFS-6367:
--

Summary: EnumSetParam$Domain#parse fails for parameter containing more than 
one enum.  (was: DomainE extends EnumE#parse in EnumSetParam fails for 
parmater containing more than one enum.)

 EnumSetParam$Domain#parse fails for parameter containing more than one enum.
 

 Key: HDFS-6367
 URL: https://issues.apache.org/jira/browse/HDFS-6367
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Affects Versions: 3.0.0
Reporter: Yi Liu
Assignee: Yi Liu
 Fix For: 3.0.0

 Attachments: HDFS-6367.patch


 Fails because additional , 
 
 java.lang.IllegalArgumentException: No enum const class 
 org.apache.hadoop.fs.Options$Rename.,OVERWRITE
   at java.lang.Enum.valueOf(Enum.java:196)
   at 
 org.apache.hadoop.hdfs.web.resources.EnumSetParam$Domain.parse(EnumSetParam.java:85)
   at 
 org.apache.hadoop.hdfs.web.resources.RenameOptionSetParam.init(RenameOptionSetParam.java:45)
   at 
 org.apache.hadoop.hdfs.web.resources.TestParam.testRenameOptionSetParam(TestParam.java:355)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
   at 
 org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
   at 
 org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
   at 
 org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
   at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
   at 
 org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
   at 
 org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6367) EnumSetParam$Domain#parse fails for parameter containing more than one enum.


[ 
https://issues.apache.org/jira/browse/HDFS-6367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13994998#comment-13994998
 ] 

Uma Maheswara Rao G commented on HDFS-6367:
---

+1 Patch looks good to me.

 EnumSetParam$Domain#parse fails for parameter containing more than one enum.
 

 Key: HDFS-6367
 URL: https://issues.apache.org/jira/browse/HDFS-6367
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Affects Versions: 3.0.0
Reporter: Yi Liu
Assignee: Yi Liu
 Fix For: 3.0.0

 Attachments: HDFS-6367.patch


 Fails because additional , 
 
 java.lang.IllegalArgumentException: No enum const class 
 org.apache.hadoop.fs.Options$Rename.,OVERWRITE
   at java.lang.Enum.valueOf(Enum.java:196)
   at 
 org.apache.hadoop.hdfs.web.resources.EnumSetParam$Domain.parse(EnumSetParam.java:85)
   at 
 org.apache.hadoop.hdfs.web.resources.RenameOptionSetParam.init(RenameOptionSetParam.java:45)
   at 
 org.apache.hadoop.hdfs.web.resources.TestParam.testRenameOptionSetParam(TestParam.java:355)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
   at 
 org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
   at 
 org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
   at 
 org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
   at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
   at 
 org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
   at 
 org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6371) In HA setup, the standby NN should redirect WebHDFS write requests to the active NN

2014-05-12 Thread Tsz Wo Nicholas Sze (JIRA)

Tsz Wo Nicholas Sze created HDFS-6371:
-

 Summary: In HA setup, the standby NN should redirect WebHDFS write 
requests to the active NN
 Key: HDFS-6371
 URL: https://issues.apache.org/jira/browse/HDFS-6371
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Reporter: Tsz Wo Nicholas Sze
Assignee: Tsz Wo Nicholas Sze


The current WebHDFS implementation in namenode does not check its HA state -- 
it does the same thing no matter it is active or standby.

Suppose a http client talk to the standby NN via WebHDFS.  For the read 
operations, there is no problem.  For the write operations, if the operation 
requires http redirect (e.g. creating a file), it will work since the standby 
NN will also redirect the client to a DN.  When the client connect to the DN, 
the DN will fulfill the request with the active NN.  However, for the write 
operations not requiring http redirect (e.g. mkdir), the operation will fail 
with StandbyException since it will be executed with the standby NN.

There are two solutions:
# The http client could catch StandbyException and then retries with the other 
NN in this case.
# The standby NN redirects the request to the active NN.

The second solution seems better since the client does not need to know both 
active NN and standby NN.

Note that WebHdfsFileSystem is already able to handle HA failover.  The JIRA is 
for other http clients.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6326) WebHdfs ACL compatibility is broken


[ 
https://issues.apache.org/jira/browse/HDFS-6326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995148#comment-13995148
 ] 

Daryn Sharp commented on HDFS-6326:
---

I still need to look at the latest code from [~cnauroth], but [~wheat9], the 
assumption in the web UI code is completely wrong and prevents any future use 
of any upper bits in the mask.  I think avoiding the performance penalty of 
double rpc/http calls just to print a + is more important than working around 
a web bug.

 WebHdfs ACL compatibility is broken
 ---

 Key: HDFS-6326
 URL: https://issues.apache.org/jira/browse/HDFS-6326
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Affects Versions: 3.0.0, 2.4.0
Reporter: Daryn Sharp
Assignee: Chris Nauroth
Priority: Blocker
 Attachments: HDFS-6326.1.patch, HDFS-6326.2.patch, HDFS-6326.3.patch


 2.4 ACL support is completely incompatible with 2.4 webhdfs servers.  The NN 
 throws an {{IllegalArgumentException}} exception.
 {code}
 hadoop fs -ls webhdfs://nn/
 Found 21 items
 ls: Invalid value for webhdfs parameter op: No enum constant 
 org.apache.hadoop.hdfs.web.resources.GetOpParam.Op.GETACLSTATUS
 [... 20 more times...]
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6294) Use INode IDs to avoid conflicts when a file open for write is renamed


 [ 
https://issues.apache.org/jira/browse/HDFS-6294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Patrick McCabe updated HDFS-6294:
---

Attachment: HDFS-6294.005.patch

 Use INode IDs to avoid conflicts when a file open for write is renamed
 --

 Key: HDFS-6294
 URL: https://issues.apache.org/jira/browse/HDFS-6294
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 0.20.1
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
 Attachments: HDFS-6294.001.patch, HDFS-6294.002.patch, 
 HDFS-6294.003.patch, HDFS-6294.004.patch, HDFS-6294.005.patch


 Now that we have a unique INode ID for each INode, clients with files that 
 are open for write can use this unique ID rather than a file path when they 
 are requesting more blocks or closing the open file.  This will avoid 
 conflicts when a file which is open for write is renamed, and another file 
 with that name is created.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6346) Optimize OP_SET_XATTRS by persisting single Xattr entry per setXattr/removeXattr api call


[ 
https://issues.apache.org/jira/browse/HDFS-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13992960#comment-13992960
 ] 

Uma Maheswara Rao G commented on HDFS-6346:
---

Patch looks great!

I have few initial comments below:

{code}
 if (removedXAttr != null) {
fsImage.getEditLog().logRemoveXAttr(src, removedXAttr);
  }
{code}
Do you think, we need warn/log to the user that xattr attempting to remove does 
not exist?

{code}
 XAttr unprotectedRemoveXAttr(String src,
  XAttr xAttr) throws IOException {
{code}
Need format


{code}
 XAttrStorage.updateINodeXAttrs(inode, newXAttrs, snapshotId);

return existingXAttrs.size() == newXAttrs.size() ? null : xAttr;
{code}
Pls remove empty line between


When existing and updated xattrs equal then we need not even call 
updateINodeXAttr

so code can be changed from
{code}

{code}

to

{code}
 if (existingXAttrs.size() != newXAttrs.size()) {
  XAttrStorage.updateINodeXAttrs(inode, newXAttrs, snapshotId);
  return xAttr;
}
return null; {code}
?

I will continue my review tomorrow on this and post if there are any comments.

 Optimize OP_SET_XATTRS by persisting single Xattr entry per 
 setXattr/removeXattr api call
 -

 Key: HDFS-6346
 URL: https://issues.apache.org/jira/browse/HDFS-6346
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Reporter: Uma Maheswara Rao G
Assignee: Yi Liu
 Attachments: HDFS-6346.patch, editsStored


 When a new xattrs set on an Inode, it may add this with OP_SET_XATTRS and 
 let's say [user.name1:value1]
 On a next call if we set another xattrs, then it may store along with older 
 existing xattrs again. It may be like [user.name1:value1, user.name2:value2]
 So, on adding more xattrs on same Inode, that list may grow and we keep store 
 the entries of already existing name, value fairs.
 Right now we defaulted the max Xattrs on an Inode to 32 and configured. If 
 user modified it to much larger value and start setting xattrs, then edits 
 loading may create issue like my above example.
 But I didn't refer any usecase of having large number of xattrs, this is just 
 the assumption to consider a case. My biggest doubt is whether we will have 
 such real usecases to have huge xattrs on a single INode.
 So, here is a thought on having OP_SET_XATTR for each setXAttr operation to 
 be logged, When we replay them we need to consolidate. This is some initial 
 thought we can think more if others also feel we need to consider this case 
 to handle.
 Otherwise we endup storing Xattrs entries in editlog file as n(n+1)/2 where n 
 is number xattrs for a file/dir. This may be issue only when we have large 
 number configured max xattrs for inode.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6366) FsImage loading failed with RemoveXattr op


[ 
https://issues.apache.org/jira/browse/HDFS-6366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13994179#comment-13994179
 ] 

Yi Liu commented on HDFS-6366:
--

Right, Uma, in patch of HDFS-6346, we missed a break for OP_REMOVE_XATTR...

 FsImage loading failed with RemoveXattr op
 --

 Key: HDFS-6366
 URL: https://issues.apache.org/jira/browse/HDFS-6366
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
 Fix For: HDFS XAttrs (HDFS-2006)

 Attachments: HDFS-6366.patch


 Should break after the removeXattr op loading.
 Otherwise fsimage loading will fail:
 java.io.IOException: Invalid operation read OP_REMOVE_XATTR
   at 
 org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:816)
   at 
 org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:227)
   at 
 org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:136)
   at 
 org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:805)
   at 
 org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:665)
   at 
 org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:272)
   at 
 org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:902)
   at 
 org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:651)
   at 
 org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:479)
   at 
 org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:535)
   at 
 org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:694)
   at 
 org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:679)
   at 
 org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1332)
   at 
 org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:973)
   at 
 org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:854)
   at 
 org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:700)
   at org.apache.hadoop.hdfs.MiniDFSCluster.init(MiniDFSCluster.java:373)
   at 
 org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:354)
   at 
 org.apache.hadoop.hdfs.server.namenode.FSXAttrBaseTest.initCluster(FSXAttrBaseTest.java:400)
   at 
 org.apache.hadoop.hdfs.server.namenode.FSXAttrBaseTest.restart(FSXAttrBaseTest.java:418)
   at 
 org.apache.hadoop.hdfs.server.namenode.FSXAttrBaseTest.testCreateXAttr(FSXAttrBaseTest.java:121)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6357) SetXattr should persist rpcIDs for handling retrycache with Namenode restart and HA


[ 
https://issues.apache.org/jira/browse/HDFS-6357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13994196#comment-13994196
 ] 

Yi Liu commented on HDFS-6357:
--

+1, the patch looks good to me.  Thanks Uma.

 SetXattr should persist rpcIDs for handling retrycache with Namenode restart 
 and HA
 ---

 Key: HDFS-6357
 URL: https://issues.apache.org/jira/browse/HDFS-6357
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Affects Versions: HDFS XAttrs (HDFS-2006)
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
 Fix For: HDFS XAttrs (HDFS-2006)

 Attachments: HDFS-6357.patch, HDFS-6357.patch, HDFS-6357.patch


 SetXattr should persist rpcIDs for handling restart Namenode and HA scenarios



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6326) WebHdfs ACL compatibility is broken

[
https://issues.apache.org/jira/browse/HDFS-6326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995278#comment-13995278
]

Haohui Mai commented on HDFS-6326:
--

bq. the assumption in the web UI code is completely wrong and prevents any
future use of any upper bits in the mask.

The point is to demonstrate how other clients of webhdfs can potentially depend
on this behavior. Though the bug is within our reach and can be fixed by
ourselves, the webhdfs protocol is intended to be implemented by 3rd-party
clients which are totally out of our control. I have no problem to declare this
is a wrong assumption but the sad fact is that 3rd-party clients might depend
on it.

bq. I think avoiding the performance penalty of double rpc/http calls just to
print a + is more important than working around a web bug.

Having the bit obviously makes implementing ls / web ui much easier, but it
comes with a performance cost on the critical path. We've implemented this
approach before and reverted it to the current state, as Chris mentioned in
earlier comments.

{{ls}} is not in the critical path, but adding a new field into the
{{FileStatus}} response is. Provided that the {{FileStatus}} field is
relatively short (~200 bytes), adding a field for ACL increases the network
traffic for listing files by 5%. The overhead is indeed significant as listing
files is the first step of distcp which is done within a single thread. We've
seen use cases that this step can take more than 7 hours.

Again having an ACL bit makes things easier to implement, but failing to
justify the performance concerns leads Chris and I to decide to keep ACL out of
the critical path, and to move the complexity to ls / web UI.

Maybe I don't understand what you're proposing, can you propose your changes
more concretely, just like what Chris did in
https://issues.apache.org/jira/browse/HDFS-6326?focusedCommentId=13991788page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13991788

WebHdfs ACL compatibility is broken
---

Key: HDFS-6326
URL: https://issues.apache.org/jira/browse/HDFS-6326
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 3.0.0, 2.4.0
Reporter: Daryn Sharp
Assignee: Chris Nauroth
Priority: Blocker
Attachments: HDFS-6326.1.patch, HDFS-6326.2.patch, HDFS-6326.3.patch

2.4 ACL support is completely incompatible with 2.4 webhdfs servers. The NN
throws an {{IllegalArgumentException}} exception.
{code}
hadoop fs -ls webhdfs://nn/
Found 21 items
ls: Invalid value for webhdfs parameter op: No enum constant
org.apache.hadoop.hdfs.web.resources.GetOpParam.Op.GETACLSTATUS
[... 20 more times...]
{code}

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6230) Expose upgrade status through NameNode web UI


 [ 
https://issues.apache.org/jira/browse/HDFS-6230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai updated HDFS-6230:
-

   Resolution: Fixed
Fix Version/s: 2.5.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

I've committed the patch to trunk and branch-2. Thanks [~mitdesai] for the 
contribution.

 Expose upgrade status through NameNode web UI
 -

 Key: HDFS-6230
 URL: https://issues.apache.org/jira/browse/HDFS-6230
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 2.4.0
Reporter: Arpit Agarwal
Assignee: Mit Desai
 Fix For: 2.5.0

 Attachments: HDFS-6230-NoUpgradesInProgress.png, 
 HDFS-6230-UpgradeInProgress.jpg, HDFS-6230.patch, HDFS-6230.patch


 The NameNode web UI does not show upgrade information anymore. Hadoop 2.0 
 also does not have the _hadoop dfsadmin -upgradeProgress_ command to check 
 the upgrade status.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6361) TestIdUserGroup.testUserUpdateSetting failed due to out of range nfsnobody Id


[ 
https://issues.apache.org/jira/browse/HDFS-6361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995353#comment-13995353
 ] 

Colin Patrick McCabe commented on HDFS-6361:


Some background would probably be helpful here.  On Linux, user IDs (UIDs) are 
32 bits.  This convention goes all the way down into the kernel.

Typically they are treated as unsigned 32-bit numbers.  So you'll find code 
like this in {{bits/typesizes.h}}:
{code}
bits/typesizes.h:#define __UID_T_TYPE   __U32_TYPE
{code}

Java, of course, does not have unsigned numbers.  So we have to represent 
numbers like 4294967295 as -1 in Java.

{code}
However, the effect of returning -2 from getUid and getGid for 
+   * nfsnobody is yet to be understood
{code}

What is it that we have yet to understand?  We better figure it out before we 
commit this code.

{code}
+  private static Integer parseId(final String idStr) {
+String tmpStr = idStr;
+if (idStr.equals(4294967294)) {
+  tmpStr = -2;
+}
+return Integer.valueOf(tmpStr);
+  }
{code}

I don't see a reason to special-case -2.  We should treat all large integers  
0x7fff as negatives.  If Java had unsigned numbers, we could just use an 
unsigned 32 bit int... but it doesn't, so we can just use a signed int.

Basically, for the purposes of user IDs, -2 and 4294967294 are the same... at 
least on Linux.

{code}
Another option to the patch I provided is to use something like a 64 bit 
integer for ID, but the would be wide scope,
and I suspect it's going to be an overkill.
{code}

The Linux kernel itself doesn't use 64-bit numbers for UIDs.  I'm not aware of 
any kernels which do.  So I don't see a benefit to this.

 TestIdUserGroup.testUserUpdateSetting failed due to out of range nfsnobody Id
 -

 Key: HDFS-6361
 URL: https://issues.apache.org/jira/browse/HDFS-6361
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: nfs
Affects Versions: 2.4.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
 Attachments: HDFS-6361.001.patch


 The following error happens pretty often:
 org.apache.hadoop.nfs.nfs3.TestIdUserGroup.testUserUpdateSetting
 Failing for the past 1 build (Since Unstable#61 )
 Took 0.1 sec.
 add description
 Error Message
 For input string: 4294967294
 Stacktrace
 java.lang.NumberFormatException: For input string: 4294967294
   at 
 java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
   at java.lang.Integer.parseInt(Integer.java:495)
   at java.lang.Integer.valueOf(Integer.java:582)
   at 
 org.apache.hadoop.nfs.nfs3.IdUserGroup.updateMapInternal(IdUserGroup.java:137)
   at 
 org.apache.hadoop.nfs.nfs3.IdUserGroup.updateMaps(IdUserGroup.java:188)
   at org.apache.hadoop.nfs.nfs3.IdUserGroup.init(IdUserGroup.java:60)
   at 
 org.apache.hadoop.nfs.nfs3.TestIdUserGroup.testUserUpdateSetting(TestIdUserGroup.java:71)
 Standard Output
 log4j:WARN No appenders could be found for logger 
 (org.apache.hadoop.nfs.nfs3.IdUserGroup).
 log4j:WARN Please initialize the log4j system properly.
 log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
 info.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6326) WebHdfs ACL compatibility is broken


[ 
https://issues.apache.org/jira/browse/HDFS-6326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995256#comment-13995256
 ] 

Daryn Sharp commented on HDFS-6326:
---

I was going to comment on the lack of bitwise and after the shift.  :)

Other comments/questions:
* Bits 10  11 are for set-uid/gid so we should probably use bit 12 to be safe.
* Since the acl bit is intended to be invisible to the client, shouldn't it be 
ignored in {{FSPermission#equals}}?
* Same for {{FsPermission#toString}}, should probably be left to FsShell or 
others to add the +.
* In {{AclCommands#processPath}}, is it guaranteed (probably yes?) that the 
owner/group/etc will be identical in the acl vs. file status?  If no, maybe the 
first line should be changed to call getAclStatus if the bit is set, and most 
of the subsequent code left alone?
* Should we avoid changing {{PBHelper. convert(FsPermission)}} and just let 
{{FsAclPermission#toShort}} serialize the bit?
* Does {{PBHelper.convert(FsPermissionProto)}} need to be changed?  Assuming 
it's only used to decode incoming permissions in a rpc call where we don't care 
about the bit?


 WebHdfs ACL compatibility is broken
 ---

 Key: HDFS-6326
 URL: https://issues.apache.org/jira/browse/HDFS-6326
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Affects Versions: 3.0.0, 2.4.0
Reporter: Daryn Sharp
Assignee: Chris Nauroth
Priority: Blocker
 Attachments: HDFS-6326.1.patch, HDFS-6326.2.patch, HDFS-6326.3.patch


 2.4 ACL support is completely incompatible with 2.4 webhdfs servers.  The NN 
 throws an {{IllegalArgumentException}} exception.
 {code}
 hadoop fs -ls webhdfs://nn/
 Found 21 items
 ls: Invalid value for webhdfs parameter op: No enum constant 
 org.apache.hadoop.hdfs.web.resources.GetOpParam.Op.GETACLSTATUS
 [... 20 more times...]
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6355) Fix divide-by-zero, improper use of wall-clock time in BlockPoolSliceScanner


[ 
https://issues.apache.org/jira/browse/HDFS-6355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995358#comment-13995358
 ] 

Colin Patrick McCabe commented on HDFS-6355:


bq. Here since we already use monotonicNow(), which is relative to jvm startup, 
I don't think we will get a negative value here and hence no need to take a 
max(..).

The issue is that {{currentPeriodStart}} is not going to increase over the 
course of a scan, but {{Time.monotonicNow}} will.  So eventually, that quantity 
could become negative.  This is certainly a corner case (a very, very slow 
scan?) but it seems better to handle it.

bq. +1

thanks, will commit later today if there's no more comments

 Fix divide-by-zero, improper use of wall-clock time in BlockPoolSliceScanner
 

 Key: HDFS-6355
 URL: https://issues.apache.org/jira/browse/HDFS-6355
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
 Attachments: HDFS-6355.001.patch


 BlockPoolSliceScanner uses {{Time.now}} to calculate an interval.  But this 
 is incorrect, since if the wall-clock time changes, we will end up setting 
 the scan periods to a shorter or longer time than we configured.
 There is also a case where we may divide by zero if we get unlucky, because 
 we calculate an interval and divide by it, without checking whether the 
 interval is 0 milliseconds.  This would produce an {{ArithmeticException}} 
 since we are using longs.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-4437) Clients should not retain leases on moved files


[ 
https://issues.apache.org/jira/browse/HDFS-4437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995374#comment-13995374
 ] 

Colin Patrick McCabe commented on HDFS-4437:


Marking this as a dupe of HDFS-6294, which addressed the issues we had with 
moving files that were open for write.  Feel free to reopen if there's 
something more we should do here.

 Clients should not retain leases on moved files
 ---

 Key: HDFS-4437
 URL: https://issues.apache.org/jira/browse/HDFS-4437
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Priority: Critical

 Moving open files and/or parent directories of open files will rewrite the 
 leases to the new path.  The client is not notified so future stream 
 operations will fail.  However, as long as the client keeps its lease active 
 it will have a lock on the file in its new location.  This is not good for 
 a daemon.
 Leases should be released after the file is moved.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6293) Issues with OIV processing PB-based fsimages

[
https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995335#comment-13995335
]

Haohui Mai commented on HDFS-6293:
--

The patch looks good to me. One nit is that can you please mark the class that
are brought back deprecated?

+1 once it is addressed.

Issues with OIV processing PB-based fsimages

Key: HDFS-6293
URL: https://issues.apache.org/jira/browse/HDFS-6293
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Kihwal Lee
Assignee: Haohui Mai
Priority: Blocker
Attachments: HDFS-6293.000.patch, HDFS-6293.001.patch,
HDFS-6293.002-save-deprecated-fsimage.patch,
HDFS-6293_sbn_ckpt_retention.patch,
HDFS-6293_sbn_ckpt_retention_oiv_legacy.patch, Heap Histogram.html

There are issues with OIV when processing fsimages in protobuf.
Due to the internal layout changes introduced by the protobuf-based fsimage,
OIV consumes excessive amount of memory. We have tested with a fsimage with
about 140M files/directories. The peak heap usage when processing this image
in pre-protobuf (i.e. pre-2.4.0) format was about 350MB. After converting
the image to the protobuf format on 2.4.0, OIV would OOM even with 80GB of
heap (max new size was 1GB). It should be possible to process any image with
the default heap size of 1.5GB.
Another issue is the complete change of format/content in OIV's XML output.
I also noticed that the secret manager section has no tokens while there were
unexpired tokens in the original image (pre-2.4.0). I did not check whether
they were also missing in the new pb fsimage.

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6294) Use INode IDs to avoid conflicts when a file open for write is renamed

2014-05-12 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-6294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995399#comment-13995399
 ] 

Hudson commented on HDFS-6294:
--

SUCCESS: Integrated in Hadoop-trunk-Commit #5603 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/5603/])
HDFS-6294. Use INode IDs to avoid conflicts when a file open for write is 
renamed (cmccabe) (cmccabe: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1593634)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/LeaseRenewer.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodesInPath.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSClientAdapter.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAbandonBlock.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend3.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCreation.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLease.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLeaseRenewer.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestPersistBlocks.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRenameWhileOpen.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSetrepDecreasing.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSetrepIncreasing.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestMetaSave.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHASafeMode.java


 Use INode IDs to avoid conflicts when a file open for write is renamed
 --

 Key: HDFS-6294
 URL: https://issues.apache.org/jira/browse/HDFS-6294
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 0.20.1
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
 Fix For: 2.5.0

 Attachments: HDFS-6294.001.patch, HDFS-6294.002.patch, 
 HDFS-6294.003.patch, HDFS-6294.004.patch, HDFS-6294.005.patch, 
 HDFS-6294.005.patch


 Now that we have a unique INode ID for each INode, clients with files that 
 are open for write can use this unique ID rather than a file path when they 
 are requesting more blocks or closing the open file.  This will avoid 
 conflicts when a file which is open for write is renamed, and another file 
 with that name is created.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6351) Command hdfs dfs -rm -r can't remove empty directory


[ 
https://issues.apache.org/jira/browse/HDFS-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995296#comment-13995296
 ] 

Andrew Wang commented on HDFS-6351:
---

I think these are both known flakes, and the fact that identical patches have 
produced different test runs also supports this hypothesis. I'll commit this 
shortly.

 Command hdfs dfs -rm -r can't remove empty directory
 --

 Key: HDFS-6351
 URL: https://issues.apache.org/jira/browse/HDFS-6351
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: hdfs-client
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
 Attachments: HDFS-6351.001.patch, HDFS-6351.002.patch, 
 HDFS-6351.002.patch


 This JIRA is fo the rmr part of the issue reported in HDFS-6165.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6326) WebHdfs ACL compatibility is broken


[ 
https://issues.apache.org/jira/browse/HDFS-6326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995430#comment-13995430
 ] 

Chris Nauroth commented on HDFS-6326:
-

Daryn, thank you for looking.

bq. I was going to comment on the lack of bitwise and after the shift.  :-)

Darn, there is no hiding my shame.  :-)

bq. Bits 10  11 are for set-uid/gid so we should probably use bit 12 to be 
safe.

Yes, we can use bit 12.

bq. Since the acl bit is intended to be invisible to the client, shouldn't it 
be ignored in {{FSPermission#equals}}?

OK, let's do that.  I'll re-raise my concern that this whole business of hiding 
the bit causes confusion in the client-side API.  Now, we'll have a situation 
where 2 instances can appear equal even though one has the ACL bit and the 
other doesn't.  I suppose it's best that we're at least consistent in ignoring 
it, so let's remove it from {{equals}}.

bq. Same for {{FsPermission#toString}}, should probably be left to {{FsShell}} 
or others to add the +.

Yes, same as above.

bq. In {{AclCommands#processPath}}, is it guaranteed (probably yes?) that the 
owner/group/etc will be identical in the acl vs. file status?

Yes, this is guaranteed, barring race conditions involving a concurrent process 
running chown interleaved between the {{getFileInfo}} and the {{getAclStatus}} 
RPCs.

bq. Should we avoid changing {{PBHelper. convert(FsPermission)}} and just let 
{{FsAclPermission#toShort}} serialize the bit?

Yes, we can do that.

bq. Does {{PBHelper.convert(FsPermissionProto)}} need to be changed? Assuming 
it's only used to decode incoming permissions in a rpc call where we don't care 
about the bit?

This change is required so that the shell (and others) can get a {{true}} 
return from {{FsPermission#getAclBit}} via the {{FsAclPermission}} subclass.  
The patch is hiding the bit from the 16-bit representation seen by callers of 
{{FsPermission#toShort}}, but we need to preserve the information somehow.  The 
PB conversion layer seems to be the only remaining choice, but let me know if 
you had another idea in mind.

 WebHdfs ACL compatibility is broken
 ---

 Key: HDFS-6326
 URL: https://issues.apache.org/jira/browse/HDFS-6326
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Affects Versions: 3.0.0, 2.4.0
Reporter: Daryn Sharp
Assignee: Chris Nauroth
Priority: Blocker
 Attachments: HDFS-6326.1.patch, HDFS-6326.2.patch, HDFS-6326.3.patch


 2.4 ACL support is completely incompatible with 2.4 webhdfs servers.  The NN 
 throws an {{IllegalArgumentException}} exception.
 {code}
 hadoop fs -ls webhdfs://nn/
 Found 21 items
 ls: Invalid value for webhdfs parameter op: No enum constant 
 org.apache.hadoop.hdfs.web.resources.GetOpParam.Op.GETACLSTATUS
 [... 20 more times...]
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Reopened] (HDFS-6134) Transparent data at rest encryption

2014-05-12 Thread Alejandro Abdelnur (JIRA)

[
https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Alejandro Abdelnur reopened HDFS-6134:
--

[cross-posting with HADOOP-10150]

Reopening HDFS-6134

After some offline discussions with Yi, Tianyou, ATM, Todd, Andrew and Charles
we think is makes more sense to implement encryption for HDFS directly into the
DistributedFileSystem client and to use CryptoFileSystem support encryption for
FileSystems that don’t support native encryption.

The reasons for this change of course are:

* If we want to we add support for HDFS transparent compression, the
compression should be done before the encryption (implying less entropy). If
compression is to be handled by HDFS DistributedFileSystem, then the encryption
has to be handled afterwards (in the write path).

* The proposed CryptoSupport abstraction significantly complicates the
implementation of CryptoFileSystem and the wiring in HDFS FileSystem client.

* Building it directly into HDFS FileSystem client may allow us to avoid an
extra copy of data.

Because of this, the idea is now:

* A common set of Crypto Input/Output streams. They would be used by
CryptoFileSystem, HDFS encryption, MapReduce intermediate data and spills. Note
we cannot use the JDK Cipher Input/Output streams directly because we need to
support the additional interfaces that the Hadoop FileSystem streams implement
(Seekable, PositionedReadable, ByteBufferReadable, HasFileDescriptor,
CanSetDropBehind, CanSetReadahead, HasEnhancedByteBufferAccess, Syncable,
CanSetDropBehind).

* CryptoFileSystem.
To support encryption in arbitrary FileSystems.

* HDFS client encryption. To support transparent HDFS encryption.

Both CryptoFilesystem and HDFS client encryption implementations would be built
using the Crypto Input/Output streams, xAttributes and KeyProvider API.

Transparent data at rest encryption
---

Key: HDFS-6134
URL: https://issues.apache.org/jira/browse/HDFS-6134
Project: Hadoop HDFS
Issue Type: New Feature
Components: security
Affects Versions: 2.3.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Attachments: HDFSDataAtRestEncryption.pdf

Because of privacy and security regulations, for many industries, sensitive
data at rest must be in encrypted form. For example: the healthcare industry
(HIPAA regulations), the card payment industry (PCI DSS regulations) or the
US government (FISMA regulations).
This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can
be used transparently by any application accessing HDFS via Hadoop Filesystem
Java API, Hadoop libhdfs C library, or WebHDFS REST API.
The resulting implementation should be able to be used in compliance with
different regulation requirements.

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6351) Command hdfs dfs -rm -r can't remove empty directory


 [ 
https://issues.apache.org/jira/browse/HDFS-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang updated HDFS-6351:
--

  Resolution: Fixed
   Fix Version/s: 2.5.0
Target Version/s: 2.5.0
  Status: Resolved  (was: Patch Available)

Committed to trunk and branch-2. Thanks again Yongjun.

 Command hdfs dfs -rm -r can't remove empty directory
 --

 Key: HDFS-6351
 URL: https://issues.apache.org/jira/browse/HDFS-6351
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: hdfs-client
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
 Fix For: 2.5.0

 Attachments: HDFS-6351.001.patch, HDFS-6351.002.patch, 
 HDFS-6351.002.patch


 This JIRA is fo the rmr part of the issue reported in HDFS-6165.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6230) Expose upgrade status through NameNode web UI

2014-05-12 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-6230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995400#comment-13995400
 ] 

Hudson commented on HDFS-6230:
--

SUCCESS: Integrated in Hadoop-trunk-Commit #5603 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/5603/])
HDFS-6230. Expose upgrade status through NameNode web UI. Contributed by Mit 
Desai. (wheat9: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1594040)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html


 Expose upgrade status through NameNode web UI
 -

 Key: HDFS-6230
 URL: https://issues.apache.org/jira/browse/HDFS-6230
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 2.4.0
Reporter: Arpit Agarwal
Assignee: Mit Desai
 Fix For: 2.5.0

 Attachments: HDFS-6230-NoUpgradesInProgress.png, 
 HDFS-6230-UpgradeInProgress.jpg, HDFS-6230.patch, HDFS-6230.patch


 The NameNode web UI does not show upgrade information anymore. Hadoop 2.0 
 also does not have the _hadoop dfsadmin -upgradeProgress_ command to check 
 the upgrade status.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6325) Append should fail if the last block has insufficient number of replicas


 [ 
https://issues.apache.org/jira/browse/HDFS-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Keith Pak updated HDFS-6325:


Attachment: HDFS-6325.patch

 Append should fail if the last block has insufficient number of replicas
 

 Key: HDFS-6325
 URL: https://issues.apache.org/jira/browse/HDFS-6325
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 2.2.0
Reporter: Konstantin Shvachko
Assignee: Keith Pak
 Attachments: HDFS-6325.patch, HDFS-6325.patch, HDFS-6325_test.patch, 
 appendTest.patch


 Currently append() succeeds on a file with the last block that has no 
 replicas. But the subsequent updatePipeline() fails as there are no replicas 
 with the exception Unable to retrieve blocks locations for last block. This 
 leaves the file unclosed, and others can not do anything with it until its 
 lease expires.
 The solution is to check replicas of the last block on the NameNode and fail 
 during append() rather than during updatePipeline().
 How many replicas should be present before NN allows to append? I see two 
 options:
 # min-replication: allow append if the last block is minimally replicated (1 
 by default)
 # full-replication: allow append if the last block is fully replicated (3 by 
 default)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6372) Handle setXattr rpcIDs for OfflineEditsViewer


 [ 
https://issues.apache.org/jira/browse/HDFS-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Uma Maheswara Rao G updated HDFS-6372:
--

Attachment: editsStored
HDFS-6372.patch

Attaching the patch for fixing the issues. I took advantage of this JIRA to fix 
the retrycachewithHA testcase.

 Handle setXattr rpcIDs for OfflineEditsViewer
 -

 Key: HDFS-6372
 URL: https://issues.apache.org/jira/browse/HDFS-6372
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Affects Versions: HDFS XAttrs (HDFS-2006)
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
 Attachments: HDFS-6372.patch, editsStored


 toXml and fromXml methods in SetXattrOp should store and read the rpcids.
 This JIRA can also fix the failures of 
 TestOfflineEditsViewer
 TestRetryCacheWithHA 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-5522) Datanode disk error check may be incorrectly skipped


[ 
https://issues.apache.org/jira/browse/HDFS-5522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995476#comment-13995476
 ] 

Kihwal Lee commented on HDFS-5522:
--

+ 1 lgtm

 Datanode disk error check may be incorrectly skipped
 

 Key: HDFS-5522
 URL: https://issues.apache.org/jira/browse/HDFS-5522
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 0.23.9, 2.2.0
Reporter: Kihwal Lee
Assignee: Rushabh S Shah
 Attachments: HDFS-5522-v2.patch, HDFS-5522-v3.patch, HDFS-5522.patch


 After HDFS-4581 and HDFS-4699, {{checkDiskError()}} is not called when 
 network errors occur during processing data node requests.  This appears to 
 create problems when a disk is having problems, but not failing I/O soon. 
 If I/O hangs for a long time, network read/write may timeout first and the 
 peer may close the connection. Although the error was caused by a faulty 
 local disk, disk check is not being carried out in this case. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6362) InvalidateBlocks is inconsistent in usage of DatanodeUuid and StorageID


[ 
https://issues.apache.org/jira/browse/HDFS-6362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13993497#comment-13993497
 ] 

Hadoop QA commented on HDFS-6362:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12644077/HDFS-6362.03.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-hdfs-project/hadoop-hdfs.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HDFS-Build/6866//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6866//console

This message is automatically generated.

 InvalidateBlocks is inconsistent in usage of DatanodeUuid and StorageID
 ---

 Key: HDFS-6362
 URL: https://issues.apache.org/jira/browse/HDFS-6362
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 2.4.0
Reporter: Arpit Agarwal
Assignee: Arpit Agarwal
Priority: Blocker
 Attachments: HDFS-6362.01.patch, HDFS-6362.02.patch, 
 HDFS-6362.03.patch


 {{InvalidateBlocks}} must consistently use datanodeUuid as the key. e.g. add 
 and remove functions use datanode UUID and storage ID.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6326) WebHdfs ACL compatibility is broken


[ 
https://issues.apache.org/jira/browse/HDFS-6326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995227#comment-13995227
 ] 

Chris Nauroth commented on HDFS-6326:
-

bq. ...the assumption in the web UI code is completely wrong and prevents any 
future use of any upper bits in the mask.

+1

I'll provide a new version of the patch that includes a fix for the web UI bug. 
 BTW, my v3 patch has a small bug in the bitwise operations to check the ACL 
bit.  I'll fix that in the next patch too.

 WebHdfs ACL compatibility is broken
 ---

 Key: HDFS-6326
 URL: https://issues.apache.org/jira/browse/HDFS-6326
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Affects Versions: 3.0.0, 2.4.0
Reporter: Daryn Sharp
Assignee: Chris Nauroth
Priority: Blocker
 Attachments: HDFS-6326.1.patch, HDFS-6326.2.patch, HDFS-6326.3.patch


 2.4 ACL support is completely incompatible with 2.4 webhdfs servers.  The NN 
 throws an {{IllegalArgumentException}} exception.
 {code}
 hadoop fs -ls webhdfs://nn/
 Found 21 items
 ls: Invalid value for webhdfs parameter op: No enum constant 
 org.apache.hadoop.hdfs.web.resources.GetOpParam.Op.GETACLSTATUS
 [... 20 more times...]
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6351) Command hdfs dfs -rm -r can't remove empty directory

2014-05-12 Thread Yongjun Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995348#comment-13995348
 ] 

Yongjun Zhang commented on HDFS-6351:
-

Thanks a lot Andrew!  And many thanks to the folks who helped reviewing and 
discussion!
 


 Command hdfs dfs -rm -r can't remove empty directory
 --

 Key: HDFS-6351
 URL: https://issues.apache.org/jira/browse/HDFS-6351
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: hdfs-client
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
 Fix For: 2.5.0

 Attachments: HDFS-6351.001.patch, HDFS-6351.002.patch, 
 HDFS-6351.002.patch


 This JIRA is fo the rmr part of the issue reported in HDFS-6165.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6293) Issues with OIV processing PB-based fsimages

2014-05-12 Thread Akira AJISAKA (JIRA)

[
https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995480#comment-13995480
]

Akira AJISAKA commented on HDFS-6293:
-

The patch looks good to me. Two comments about oiv_legacy:
# -maxSize and -step options will fail (originally reported by HDFS-5866)
# missing '\n' in the usage (HDFS-5864)

I'm thinking the main purpose of the issue is to bring back the legacy code to
keep backward compatibility, so we can fix them in a separate jira.

Issues with OIV processing PB-based fsimages

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6134) Transparent data at rest encryption

2014-05-12 Thread Owen O'Malley (JIRA)

[
https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995588#comment-13995588
]

Owen O'Malley commented on HDFS-6134:
-

What are the use cases this is trying to address? What are the attacks?

Do users or administrators set the encryption?

Can different directories have different keys or is it one key for the entire
filesystem?

When you rename a directory does it need to be re-encrypted?

How are backups handled? Does it require the encryption key? What is the
performance impact on distcp when not using native libraries?

For release in the Hadoop 2.x line, you need to preserve both forward and
backwards wire compatibility. How do you plan to address that?

It seems that the additional datanode and client complexity is prohibitive.
Making changes to the HDFS write and read pipeline is extremely touchy.

Transparent data at rest encryption
---

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6326) WebHdfs ACL compatibility is broken


[ 
https://issues.apache.org/jira/browse/HDFS-6326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995650#comment-13995650
 ] 

Hadoop QA commented on HDFS-6326:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12644230/HDFS-6326.3.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 4 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

  org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HDFS-Build/6875//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6875//console

This message is automatically generated.

 WebHdfs ACL compatibility is broken
 ---

 Key: HDFS-6326
 URL: https://issues.apache.org/jira/browse/HDFS-6326
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Affects Versions: 3.0.0, 2.4.0
Reporter: Daryn Sharp
Assignee: Chris Nauroth
Priority: Blocker
 Attachments: HDFS-6326.1.patch, HDFS-6326.2.patch, HDFS-6326.3.patch


 2.4 ACL support is completely incompatible with 2.4 webhdfs servers.  The NN 
 throws an {{IllegalArgumentException}} exception.
 {code}
 hadoop fs -ls webhdfs://nn/
 Found 21 items
 ls: Invalid value for webhdfs parameter op: No enum constant 
 org.apache.hadoop.hdfs.web.resources.GetOpParam.Op.GETACLSTATUS
 [... 20 more times...]
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6375) Listing extended attributes with the search permission

Andrew Wang created HDFS-6375:
-

 Summary: Listing extended attributes with the search permission
 Key: HDFS-6375
 URL: https://issues.apache.org/jira/browse/HDFS-6375
 Project: Hadoop HDFS
  Issue Type: Sub-task
Affects Versions: HDFS XAttrs (HDFS-2006)
Reporter: Andrew Wang


From the attr(5) manpage:

{noformat}
   Users with search access to a file or directory may retrieve a list  of
   attribute names defined for that file or directory.
{noformat}

This is like doing {{getfattr}} without the {{-d}} flag, which we currently 
don't support.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6375) Listing extended attributes with the search permission


[ 
https://issues.apache.org/jira/browse/HDFS-6375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995652#comment-13995652
 ] 

Andrew Wang commented on HDFS-6375:
---

 We don't necessarily need to add full-on support, but we should at least 
future-proof the getXAttr API with a flag EnumSet.

 Listing extended attributes with the search permission
 --

 Key: HDFS-6375
 URL: https://issues.apache.org/jira/browse/HDFS-6375
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Affects Versions: HDFS XAttrs (HDFS-2006)
Reporter: Andrew Wang

 From the attr(5) manpage:
 {noformat}
Users with search access to a file or directory may retrieve a list  of
attribute names defined for that file or directory.
 {noformat}
 This is like doing {{getfattr}} without the {{-d}} flag, which we currently 
 don't support.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6193) HftpFileSystem open should throw FileNotFoundException for non-existing paths


 [ 
https://issues.apache.org/jira/browse/HDFS-6193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai updated HDFS-6193:
-

Priority: Major  (was: Blocker)

 HftpFileSystem open should throw FileNotFoundException for non-existing paths
 -

 Key: HDFS-6193
 URL: https://issues.apache.org/jira/browse/HDFS-6193
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
 Attachments: HDFS-6193-branch-2.4.0.v01.patch, 
 HDFS-6193-branch-2.4.v02.patch


 WebHdfsFileSystem.open and HftpFileSystem.open incorrectly handles 
 non-existing paths. 
 - 'open', does not really open anything, i.e., it does not contact the 
 server, and therefore cannot discover FileNotFound, it's deferred until next 
 read. It's counterintuitive and not how local FS or HDFS work. In POSIX you 
 get ENOENT on open. 
 [LzoInputFormat.getSplits|https://github.com/kevinweil/elephant-bird/blob/master/core/src/main/java/com/twitter/elephantbird/mapreduce/input/LzoInputFormat.java]
  is an example of the code that's broken because of this.
 - On the server side, FileDataServlet incorrectly sends SC_BAD_REQUEST 
 instead of SC_NOT_FOUND for non-exitsing paths



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6293) Issues with OIV processing PB-based fsimages

[
https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Kihwal Lee updated HDFS-6293:
-

Attachment: HDFS-6293_sbn_ckpt_retention_oiv_legacy.patch

The new patch adds the old OIV back in addition to the previous changes. It can
be used by hdfs oiv_legacy command.

Issues with OIV processing PB-based fsimages

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6283) Write end user documentation for xattrs.


[ 
https://issues.apache.org/jira/browse/HDFS-6283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995302#comment-13995302
 ] 

Andrew Wang commented on HDFS-6283:
---

Hi [~hitliuyi], would it help if I picked this one up? I think this is one of 
the last things we need to get done before calling a merge vote, and I think I 
could knock this out in a day or two.

 Write end user documentation for xattrs.
 

 Key: HDFS-6283
 URL: https://issues.apache.org/jira/browse/HDFS-6283
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: documentation
Reporter: Chris Nauroth
Assignee: Yi Liu

 Update the File System Shell documentation to cover the new getfattr and 
 setfattr commands.  If warranted, consider adding a separate dedicated page 
 for fuller discussion of the xattrs model and how the feature works in more 
 detail.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Assigned] (HDFS-6374) setXAttr should require the user to be the owner of the file or directory

2014-05-12 Thread Charles Lamb (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-6374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Charles Lamb reassigned HDFS-6374:
--

Assignee: Charles Lamb

 setXAttr should require the user to be the owner of the file or directory
 -

 Key: HDFS-6374
 URL: https://issues.apache.org/jira/browse/HDFS-6374
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Affects Versions: HDFS XAttrs (HDFS-2006)
Reporter: Andrew Wang
Assignee: Charles Lamb

 From the attr(5) manpage:
 {noformat}
For  this reason, extended user attributes are only allowed for regular
files and directories,  and  access  to  extended  user  attributes  is
restricted  to the owner and to users with appropriate capabilities for
directories with the sticky bit set (see the chmod(1) manual  page  for
an explanation of Sticky Directories).
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6351) Command hdfs dfs -rm -r can't remove empty directory


[ 
https://issues.apache.org/jira/browse/HDFS-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13993907#comment-13993907
 ] 

Hadoop QA commented on HDFS-6351:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12644120/HDFS-6351.002.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 3 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-hdfs-project/hadoop-hdfs:

  
org.apache.hadoop.hdfs.server.datanode.fsdataset.TestAvailableSpaceVolumeChoosingPolicy
  org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HDFS-Build/6867//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6867//console

This message is automatically generated.

 Command hdfs dfs -rm -r can't remove empty directory
 --

 Key: HDFS-6351
 URL: https://issues.apache.org/jira/browse/HDFS-6351
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: hdfs-client
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
 Attachments: HDFS-6351.001.patch, HDFS-6351.002.patch, 
 HDFS-6351.002.patch


 This JIRA is fo the rmr part of the issue reported in HDFS-6165.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-5522) Datanode disk error check may be incorrectly skipped


 [ 
https://issues.apache.org/jira/browse/HDFS-5522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kihwal Lee updated HDFS-5522:
-

   Resolution: Fixed
Fix Version/s: 2.5.0
   3.0.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Thanks for working on this, Rushabh. I've committed this to branch-2 and trunk. 
 This feature makes disk check asynchronous, so handlers(DataXceiver) won't get 
blocked while checking disks.  

 Datanode disk error check may be incorrectly skipped
 

 Key: HDFS-5522
 URL: https://issues.apache.org/jira/browse/HDFS-5522
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 0.23.9, 2.2.0
Reporter: Kihwal Lee
Assignee: Rushabh S Shah
 Fix For: 3.0.0, 2.5.0

 Attachments: HDFS-5522-v2.patch, HDFS-5522-v3.patch, HDFS-5522.patch


 After HDFS-4581 and HDFS-4699, {{checkDiskError()}} is not called when 
 network errors occur during processing data node requests.  This appears to 
 create problems when a disk is having problems, but not failing I/O soon. 
 If I/O hangs for a long time, network read/write may timeout first and the 
 peer may close the connection. Although the error was caused by a faulty 
 local disk, disk check is not being carried out in this case. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6376) Distcp data between two HA clusters requires another configuration

2014-05-12 Thread Dave Marion (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-6376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dave Marion updated HDFS-6376:
--

Attachment: HDFS-6376-patch-1.patch

Patch 1 adds a new hdfs-site property which allows the user to specify which 
nameservices are *not* part of the cluster. This allows configuration 
information to be supplied for other clusters so that clients can resolve their 
locations.

 Distcp data between two HA clusters requires another configuration
 --

 Key: HDFS-6376
 URL: https://issues.apache.org/jira/browse/HDFS-6376
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: federation
Affects Versions: 2.3.0, 2.4.0
 Environment: CDH 5.0 (Apache 2.3.0 + some patches)
Reporter: Dave Marion
 Attachments: HDFS-6376-patch-1.patch


 User has to create a third set of configuration files for distcp when 
 transferring data between two HA clusters.
 Consider the scenario in [1]. You cannot put all of the required properties 
 in core-site.xml and hdfs-site.xml for the client to resolve the location of 
 both active namenodes. If you do, then the datanodes from cluster A may join 
 cluster B. I can not find a configuration option that tells the datanodes to 
 federate blocks for only one of the clusters in the configuration.
 [1] 
 http://mail-archives.apache.org/mod_mbox/hadoop-user/201404.mbox/%3CBAY172-W2133964E0C283968C161DD1520%40phx.gbl%3E



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6186) Pause deletion of blocks when the namenode starts up

2014-05-12 Thread Jing Zhao (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-6186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jing Zhao updated HDFS-6186:


Status: Patch Available  (was: Open)

 Pause deletion of blocks when the namenode starts up
 

 Key: HDFS-6186
 URL: https://issues.apache.org/jira/browse/HDFS-6186
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Reporter: Suresh Srinivas
Assignee: Jing Zhao
 Attachments: HDFS-6186.000.patch, HDFS-6186.002.patch


 HDFS namenode can delete blocks very quickly, given the deletion happens as a 
 parallel operation spread across many datanodes. One of the frequent 
 anxieties I see is that a lot of data can be deleted very quickly, when a 
 cluster is brought up, especially when one of the storage directories has 
 failed and namenode metadata was copied from another storage. Copying wrong 
 metadata would results in some of the newer files (if old metadata was 
 copied) being deleted along with their blocks. 
 HDFS-5986 now captures the number of pending deletion block on namenode webUI 
 and JMX. I propose pausing deletion of blocks for a configured period of time 
 (default 1 hour?) after namenode comes out of safemode. This will give enough 
 time for the administrator to notice large number of pending deletion blocks 
 and take corrective action.
 Thoughts?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6358) WebHdfs DN's DFSClient should not use a retry policy


 [ 
https://issues.apache.org/jira/browse/HDFS-6358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daryn Sharp updated HDFS-6358:
--

Target Version/s: 2.5.0

 WebHdfs DN's DFSClient should not use a retry policy
 

 Key: HDFS-6358
 URL: https://issues.apache.org/jira/browse/HDFS-6358
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Priority: Critical

 DFSClient retries on the DN are useless.  The webhdfs client is going to 
 timeout before the retries complete.  The DFSClient will also continue to run 
 until it timeouts.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6186) Pause deletion of blocks when the namenode starts up

2014-05-12 Thread Jing Zhao (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-6186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jing Zhao updated HDFS-6186:


Attachment: HDFS-6186.002.patch

Thanks for the comments, Ming and Suresh!

After an offline discussion with [~sureshms], looks like a simplified mechanism 
here can be: to delay the block deletion for a specific period of time after NN 
startup, no matter the invalid block is caused by normal file deletion or 
unknown block in block report.

Update the patch to implement the idea. The patch adds a new configuration 
property dfs.block.pending.invalidation.ms to control the delaying period, 
and simply handles all the delaying logic in InvalidateBlocks.

 Pause deletion of blocks when the namenode starts up
 

 Key: HDFS-6186
 URL: https://issues.apache.org/jira/browse/HDFS-6186
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Reporter: Suresh Srinivas
Assignee: Jing Zhao
 Attachments: HDFS-6186.000.patch, HDFS-6186.002.patch


 HDFS namenode can delete blocks very quickly, given the deletion happens as a 
 parallel operation spread across many datanodes. One of the frequent 
 anxieties I see is that a lot of data can be deleted very quickly, when a 
 cluster is brought up, especially when one of the storage directories has 
 failed and namenode metadata was copied from another storage. Copying wrong 
 metadata would results in some of the newer files (if old metadata was 
 copied) being deleted along with their blocks. 
 HDFS-5986 now captures the number of pending deletion block on namenode webUI 
 and JMX. I propose pausing deletion of blocks for a configured period of time 
 (default 1 hour?) after namenode comes out of safemode. This will give enough 
 time for the administrator to notice large number of pending deletion blocks 
 and take corrective action.
 Thoughts?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6326) WebHdfs ACL compatibility is broken


[ 
https://issues.apache.org/jira/browse/HDFS-6326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995723#comment-13995723
 ] 

Chris Nauroth commented on HDFS-6326:
-

bq. Should we avoid changing PBHelper. convert(FsPermission) and just let 
FsAclPermission#toShort serialize the bit?

Actually, I take it back.  We can't do this part.  The point of 
{{FsAclPermission}} was to hide the ACL bit from being visible in the bit 
representation.  If the subclass just overrides {{toShort}} to do this, then it 
defeats the purpose.  I'll keep this logic in {{PBHelper}}.

 WebHdfs ACL compatibility is broken
 ---

 Key: HDFS-6326
 URL: https://issues.apache.org/jira/browse/HDFS-6326
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Affects Versions: 3.0.0, 2.4.0
Reporter: Daryn Sharp
Assignee: Chris Nauroth
Priority: Blocker
 Attachments: HDFS-6326.1.patch, HDFS-6326.2.patch, HDFS-6326.3.patch


 2.4 ACL support is completely incompatible with 2.4 webhdfs servers.  The NN 
 throws an {{IllegalArgumentException}} exception.
 {code}
 hadoop fs -ls webhdfs://nn/
 Found 21 items
 ls: Invalid value for webhdfs parameter op: No enum constant 
 org.apache.hadoop.hdfs.web.resources.GetOpParam.Op.GETACLSTATUS
 [... 20 more times...]
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HDFS-6341) Clean up audit logging in FSNamesystem xattr methods


 [ 
https://issues.apache.org/jira/browse/HDFS-6341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang resolved HDFS-6341.
---

Resolution: Not a Problem

I looked at this code again, and decided that the audit logging pattern was 
reasonable. Closing as not a problem.

 Clean up audit logging in FSNamesystem xattr methods
 

 Key: HDFS-6341
 URL: https://issues.apache.org/jira/browse/HDFS-6341
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Affects Versions: HDFS XAttrs (HDFS-2006)
Reporter: Andrew Wang
Assignee: Andrew Wang
Priority: Minor

 We call logAuditEvent multiple times in each RPC handler, sometimes split 
 across different functions. It'd be cleaner if we did this once, in a 
 try/finally.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6326) WebHdfs ACL compatibility is broken


 [ 
https://issues.apache.org/jira/browse/HDFS-6326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Nauroth updated HDFS-6326:


Attachment: HDFS-6326.4.patch

Here is patch v4 incorporating the feedback mentioned in the last several 
comments.

While fixing the web UI, I also noticed that the logic for printing 't' vs. 'T' 
for the sticky bit was incorrect.  Due to the way the for loop works, this was 
always checking the sticky bit itself instead of the other execute bit.  At the 
point where the check is performed, the sticky bit is always on anyway, so it 
always chose to print 't'.  I fixed this bug too.

 WebHdfs ACL compatibility is broken
 ---

 Key: HDFS-6326
 URL: https://issues.apache.org/jira/browse/HDFS-6326
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Affects Versions: 3.0.0, 2.4.0
Reporter: Daryn Sharp
Assignee: Chris Nauroth
Priority: Blocker
 Attachments: HDFS-6326.1.patch, HDFS-6326.2.patch, HDFS-6326.3.patch, 
 HDFS-6326.4.patch


 2.4 ACL support is completely incompatible with 2.4 webhdfs servers.  The NN 
 throws an {{IllegalArgumentException}} exception.
 {code}
 hadoop fs -ls webhdfs://nn/
 Found 21 items
 ls: Invalid value for webhdfs parameter op: No enum constant 
 org.apache.hadoop.hdfs.web.resources.GetOpParam.Op.GETACLSTATUS
 [... 20 more times...]
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6326) WebHdfs ACL compatibility is broken


[ 
https://issues.apache.org/jira/browse/HDFS-6326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995775#comment-13995775
 ] 

Chris Nauroth commented on HDFS-6326:
-

While v4 is under review, I'll repeat my manual compatibility testing using a 
mix of 2.3.0 and trunk components.

 WebHdfs ACL compatibility is broken
 ---

 Key: HDFS-6326
 URL: https://issues.apache.org/jira/browse/HDFS-6326
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Affects Versions: 3.0.0, 2.4.0
Reporter: Daryn Sharp
Assignee: Chris Nauroth
Priority: Blocker
 Attachments: HDFS-6326.1.patch, HDFS-6326.2.patch, HDFS-6326.3.patch, 
 HDFS-6326.4.patch


 2.4 ACL support is completely incompatible with 2.4 webhdfs servers.  The NN 
 throws an {{IllegalArgumentException}} exception.
 {code}
 hadoop fs -ls webhdfs://nn/
 Found 21 items
 ls: Invalid value for webhdfs parameter op: No enum constant 
 org.apache.hadoop.hdfs.web.resources.GetOpParam.Op.GETACLSTATUS
 [... 20 more times...]
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6361) TestIdUserGroup.testUserUpdateSetting failed due to out of range nfsnobody Id

2014-05-12 Thread Brandon Li (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-6361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995783#comment-13995783
 ] 

Brandon Li commented on HDFS-6361:
--

So sorry for the late response. Thanks [~yzhangal] and [~cmccabe] for the 
finding the issue and giving the background description.

I agree that we should not treat -2 as a special case. Basically, user can 
configure anything in /etc/passwd or corresponding files on purpose or by 
mistake. For example, on MacOS nobody is -2 and nogroup is -1. On Linux, the 
system converts user id -3 in /etc/passwd to 4294967295. 

NFS gateway get the uid/gid from a shell command. In the result, the id string 
is from uint32. To handle the id from negative number, we can use Long to parse 
the string and then do Long.getInt() to return the correct value.

{quote}..., whether all consumers of Nfs3Util.getFileAttr() will be happy{quote}
The consumers of Nfs3Util.getFileAttr() should be OK with that since it looks 
up the id by using the name retrieved from HDFS.  Let me know if I 
misunderstood your question.




 TestIdUserGroup.testUserUpdateSetting failed due to out of range nfsnobody Id
 -

 Key: HDFS-6361
 URL: https://issues.apache.org/jira/browse/HDFS-6361
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: nfs
Affects Versions: 2.4.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
 Attachments: HDFS-6361.001.patch


 The following error happens pretty often:
 org.apache.hadoop.nfs.nfs3.TestIdUserGroup.testUserUpdateSetting
 Failing for the past 1 build (Since Unstable#61 )
 Took 0.1 sec.
 add description
 Error Message
 For input string: 4294967294
 Stacktrace
 java.lang.NumberFormatException: For input string: 4294967294
   at 
 java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
   at java.lang.Integer.parseInt(Integer.java:495)
   at java.lang.Integer.valueOf(Integer.java:582)
   at 
 org.apache.hadoop.nfs.nfs3.IdUserGroup.updateMapInternal(IdUserGroup.java:137)
   at 
 org.apache.hadoop.nfs.nfs3.IdUserGroup.updateMaps(IdUserGroup.java:188)
   at org.apache.hadoop.nfs.nfs3.IdUserGroup.init(IdUserGroup.java:60)
   at 
 org.apache.hadoop.nfs.nfs3.TestIdUserGroup.testUserUpdateSetting(TestIdUserGroup.java:71)
 Standard Output
 log4j:WARN No appenders could be found for logger 
 (org.apache.hadoop.nfs.nfs3.IdUserGroup).
 log4j:WARN Please initialize the log4j system properly.
 log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
 info.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Assigned] (HDFS-6373) Remove support for extended attributes on symlinks

2014-05-12 Thread Charles Lamb (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-6373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Charles Lamb reassigned HDFS-6373:
--

Assignee: Charles Lamb

 Remove support for extended attributes on symlinks
 --

 Key: HDFS-6373
 URL: https://issues.apache.org/jira/browse/HDFS-6373
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Reporter: Andrew Wang
Assignee: Charles Lamb

 Looking in the Linux source code, we see the following:
 http://lxr.linux.no/linux+v3.14.3/fs/xattr.c
 {code}
   60/*
   61 * In the user.* namespace, only regular files and directories 
 can have
   62 * extended attributes. For sticky directories, only the owner and
   63 * privileged users can write attributes.
   64 */
 {code}
 We should consider removing {{XAttrFeature}} from {{INodeSymlink}}.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6376) Distcp data between two HA clusters requires another configuration

[
https://issues.apache.org/jira/browse/HDFS-6376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995773#comment-13995773
]

Hadoop QA commented on HDFS-6376:
-

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment

http://issues.apache.org/jira/secure/attachment/12644500/HDFS-6376-patch-1.patch
against trunk revision .

{color:red}-1 patch{color}. The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6881//console

This message is automatically generated.

Distcp data between two HA clusters requires another configuration
--

Key: HDFS-6376
URL: https://issues.apache.org/jira/browse/HDFS-6376
Project: Hadoop HDFS
Issue Type: Bug
Components: federation
Affects Versions: 2.3.0, 2.4.0
Environment: CDH 5.0 (Apache 2.3.0 + some patches)
Reporter: Dave Marion
Attachments: HDFS-6376-patch-1.patch

User has to create a third set of configuration files for distcp when
transferring data between two HA clusters.
Consider the scenario in [1]. You cannot put all of the required properties
in core-site.xml and hdfs-site.xml for the client to resolve the location of
both active namenodes. If you do, then the datanodes from cluster A may join
cluster B. I can not find a configuration option that tells the datanodes to
federate blocks for only one of the clusters in the configuration.
[1]
http://mail-archives.apache.org/mod_mbox/hadoop-user/201404.mbox/%3CBAY172-W2133964E0C283968C161DD1520%40phx.gbl%3E

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6377) Unify xattr name and value limits into a single limit

Andrew Wang created HDFS-6377:
-

 Summary: Unify xattr name and value limits into a single limit
 Key: HDFS-6377
 URL: https://issues.apache.org/jira/browse/HDFS-6377
 Project: Hadoop HDFS
  Issue Type: Sub-task
Affects Versions: HDFS XAttrs (HDFS-2006)
Reporter: Andrew Wang


Instead of having separate limits and config options for the size of an xattr's 
name and value, let's use a single limit.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6134) Transparent data at rest encryption

2014-05-12 Thread Andrew Purtell (JIRA)

[
https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995828#comment-13995828
]

Andrew Purtell commented on HDFS-6134:
--

{quote}
Because of this, the idea is now:
- A common set of Crypto Input/Output streams. [ ... ]
- CryptoFileSystem. [ ... ]
- HDFS client encryption. [ ... ]
{quote}

I would be great if the work is structured this way. A filtering
CryptoFileSystem is needed for filesystem agnostic client side use cases, but
e.g. if we want to push compression and encryption in HBase down into HDFS
(which I think is desirable), or Hive or Pig or really any HDFS hosted Hadoop
application, then doing so is far simpler if the DFS client supports
transparent encryption directly.

Transparent data at rest encryption
---

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6369) RemoteBlockReader#available() should call FSInputChecker.available()


[ 
https://issues.apache.org/jira/browse/HDFS-6369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995791#comment-13995791
 ] 

Hadoop QA commented on HDFS-6369:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12644364/HDFS-6369.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-hdfs-project/hadoop-hdfs.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HDFS-Build/6878//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6878//console

This message is automatically generated.

 RemoteBlockReader#available() should call FSInputChecker.available()
 

 Key: HDFS-6369
 URL: https://issues.apache.org/jira/browse/HDFS-6369
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu
Assignee: Chen He
Priority: Trivial
 Attachments: HDFS-6369.patch


 Currently DFSClient.TCP_WINDOW_SIZE is directly returned.
 However, FSInputChecker.available(), in the superclass, may return value 
 lower than the constant.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6376) Distcp data between two HA clusters requires another configuration

2014-05-12 Thread Dave Marion (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-6376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dave Marion updated HDFS-6376:
--

Affects Version/s: 2.4.0
   Status: Patch Available  (was: Open)

Note that the patch was developed against CDH5 source.

 Distcp data between two HA clusters requires another configuration
 --

 Key: HDFS-6376
 URL: https://issues.apache.org/jira/browse/HDFS-6376
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: federation
Affects Versions: 2.4.0, 2.3.0
 Environment: CDH 5.0 (Apache 2.3.0 + some patches)
Reporter: Dave Marion
 Attachments: HDFS-6376-patch-1.patch


 User has to create a third set of configuration files for distcp when 
 transferring data between two HA clusters.
 Consider the scenario in [1]. You cannot put all of the required properties 
 in core-site.xml and hdfs-site.xml for the client to resolve the location of 
 both active namenodes. If you do, then the datanodes from cluster A may join 
 cluster B. I can not find a configuration option that tells the datanodes to 
 federate blocks for only one of the clusters in the configuration.
 [1] 
 http://mail-archives.apache.org/mod_mbox/hadoop-user/201404.mbox/%3CBAY172-W2133964E0C283968C161DD1520%40phx.gbl%3E



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6378) NFS: when portmap/rpcbind is not available, NFS registration should timeout instead of hanging

2014-05-12 Thread Brandon Li (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-6378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brandon Li updated HDFS-6378:
-

Description: When portmap/rpcbind is not available, NFS registration should 
timeout instead of hanging. Instead, NFS gateway should shut down automatically 
with proper error message.  (was: NFS gateway should  )

 NFS: when portmap/rpcbind is not available, NFS registration should timeout 
 instead of hanging 
 ---

 Key: HDFS-6378
 URL: https://issues.apache.org/jira/browse/HDFS-6378
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: nfs
Reporter: Brandon Li

 When portmap/rpcbind is not available, NFS registration should timeout 
 instead of hanging. Instead, NFS gateway should shut down automatically with 
 proper error message.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6255) fuse_dfs will not adhere to ACL permissions in some cases

2014-05-12 Thread Stephen Chu (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995847#comment-13995847
 ] 

Stephen Chu commented on HDFS-6255:
---

Hi Chris and Colin, thanks for looking into this. I'll try with -oallow_other 
and confirm it works.

 fuse_dfs will not adhere to ACL permissions in some cases
 -

 Key: HDFS-6255
 URL: https://issues.apache.org/jira/browse/HDFS-6255
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: fuse-dfs
Affects Versions: 3.0.0, 2.4.0
Reporter: Stephen Chu
Assignee: Chris Nauroth

 As hdfs user, I created a directory /tmp/acl_dir/ and set permissions to 700. 
 Then I set a new acl group:jenkins:rwx on /tmp/acl_dir.
 {code}
 jenkins@hdfs-vanilla-1 ~]$ hdfs dfs -getfacl /tmp/acl_dir
 # file: /tmp/acl_dir
 # owner: hdfs
 # group: supergroup
 user::rwx
 group::---
 group:jenkins:rwx
 mask::rwx
 other::---
 {code}
 Through the FsShell, the jenkins user can list /tmp/acl_dir as well as create 
 a file and directory inside.
 {code}
 [jenkins@hdfs-vanilla-1 ~]$ hdfs dfs -touchz /tmp/acl_dir/testfile1
 [jenkins@hdfs-vanilla-1 ~]$ hdfs dfs -mkdir /tmp/acl_dir/testdir1
 hdfs dfs -ls /tmp/acl[jenkins@hdfs-vanilla-1 ~]$ hdfs dfs -ls /tmp/acl_dir/
 Found 2 items
 drwxr-xr-x   - jenkins supergroup  0 2014-04-17 19:11 
 /tmp/acl_dir/testdir1
 -rw-r--r--   1 jenkins supergroup  0 2014-04-17 19:11 
 /tmp/acl_dir/testfile1
 [jenkins@hdfs-vanilla-1 ~]$ 
 {code}
 However, as the same jenkins user, when I try to cd into /tmp/acl_dir using a 
 fuse_dfs mount, I get permission denied. Same permission denied when I try to 
 create or list files.
 {code}
 [jenkins@hdfs-vanilla-1 tmp]$ ls -l
 total 16
 drwxrwx--- 4 hdfsnobody 4096 Apr 17 19:11 acl_dir
 drwx-- 2 hdfsnobody 4096 Apr 17 18:30 acl_dir_2
 drwxr-xr-x 3 mapred  nobody 4096 Mar 11 03:53 mapred
 drwxr-xr-x 4 jenkins nobody 4096 Apr 17 07:25 testcli
 -rwx-- 1 hdfsnobody0 Apr  7 17:18 tf1
 [jenkins@hdfs-vanilla-1 tmp]$ cd acl_dir
 bash: cd: acl_dir: Permission denied
 [jenkins@hdfs-vanilla-1 tmp]$ touch acl_dir/testfile2
 touch: cannot touch `acl_dir/testfile2': Permission denied
 [jenkins@hdfs-vanilla-1 tmp]$ mkdir acl_dir/testdir2
 mkdir: cannot create directory `acl_dir/testdir2': Permission denied
 [jenkins@hdfs-vanilla-1 tmp]$ 
 {code}
 The fuse_dfs debug output doesn't show any error for the above operations:
 {code}
 unique: 18, opcode: OPENDIR (27), nodeid: 2, insize: 48
unique: 18, success, outsize: 32
 unique: 19, opcode: READDIR (28), nodeid: 2, insize: 80
 readdir[0] from 0
unique: 19, success, outsize: 312
 unique: 20, opcode: GETATTR (3), nodeid: 2, insize: 56
 getattr /tmp
unique: 20, success, outsize: 120
 unique: 21, opcode: READDIR (28), nodeid: 2, insize: 80
unique: 21, success, outsize: 16
 unique: 22, opcode: RELEASEDIR (29), nodeid: 2, insize: 64
unique: 22, success, outsize: 16
 unique: 23, opcode: GETATTR (3), nodeid: 2, insize: 56
 getattr /tmp
unique: 23, success, outsize: 120
 unique: 24, opcode: GETATTR (3), nodeid: 3, insize: 56
 getattr /tmp/acl_dir
unique: 24, success, outsize: 120
 unique: 25, opcode: GETATTR (3), nodeid: 3, insize: 56
 getattr /tmp/acl_dir
unique: 25, success, outsize: 120
 unique: 26, opcode: GETATTR (3), nodeid: 3, insize: 56
 getattr /tmp/acl_dir
unique: 26, success, outsize: 120
 unique: 27, opcode: GETATTR (3), nodeid: 3, insize: 56
 getattr /tmp/acl_dir
unique: 27, success, outsize: 120
 unique: 28, opcode: GETATTR (3), nodeid: 3, insize: 56
 getattr /tmp/acl_dir
unique: 28, success, outsize: 120
 {code}
 In other scenarios, ACL permissions are enforced successfully. For example, 
 as hdfs user I create /tmp/acl_dir_2 and set permissions to 777. I then set 
 the acl user:jenkins:--- on the directory. On the fuse mount, I am not able 
 to ls, mkdir, or touch to that directory as jenkins user.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6283) Write end user documentation for xattrs.


 [ 
https://issues.apache.org/jira/browse/HDFS-6283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang updated HDFS-6283:
--

Attachment: hdfs-6283-1.patch

Patch attached. I also took the opportunity to fix some whitespace and a typo 
in XAttrCommands, and add the new parameters to hdfs-default.xml.

 Write end user documentation for xattrs.
 

 Key: HDFS-6283
 URL: https://issues.apache.org/jira/browse/HDFS-6283
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: documentation
Reporter: Chris Nauroth
Assignee: Yi Liu
 Attachments: hdfs-6283-1.patch


 Update the File System Shell documentation to cover the new getfattr and 
 setfattr commands.  If warranted, consider adding a separate dedicated page 
 for fuller discussion of the xattrs model and how the feature works in more 
 detail.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Assigned] (HDFS-6283) Write end user documentation for xattrs.


 [ 
https://issues.apache.org/jira/browse/HDFS-6283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang reassigned HDFS-6283:
-

Assignee: Andrew Wang  (was: Yi Liu)

 Write end user documentation for xattrs.
 

 Key: HDFS-6283
 URL: https://issues.apache.org/jira/browse/HDFS-6283
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: documentation
Reporter: Chris Nauroth
Assignee: Andrew Wang
 Attachments: hdfs-6283-1.patch


 Update the File System Shell documentation to cover the new getfattr and 
 setfattr commands.  If warranted, consider adding a separate dedicated page 
 for fuller discussion of the xattrs model and how the feature works in more 
 detail.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6293) Issues with OIV processing PB-based fsimages

2014-05-12 Thread Suresh Srinivas (JIRA)

[
https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995668#comment-13995668
]

Suresh Srinivas commented on HDFS-6293:
---

I agree with [~kihwal] on deprecation. Is this also going into trunk? I think
first it should be committed to trunk and then branch-2. If we are in a
position to remove it from trunk later, then it can be removed later. Thoughts?

+1 for the patch with Jenkins +1 (if it is going to trunk as well).

Issues with OIV processing PB-based fsimages

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-5682) Heterogeneous Storage phase 2 - APIs to expose Storage Types

2014-05-12 Thread Zesheng Wu (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-5682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995944#comment-13995944
 ] 

Zesheng Wu commented on HDFS-5682:
--

Thanks [~arpitagarwal], recently some of the our users have strong requirement 
of read/write SLA on HBase, and we notice that HDFS's heterogeneous storage 
feature is really interesting and helpful,  thanks for your valuable work on 
this:) 
Last week I looked into HDFS-2832 and found that we need HDFS-5682 to be 
finished before heterogeneous storage can be used, so I  asked for the progress 
of this issue. As you said this was deprioritized, I am wondering whether you 
can give a more specific plan, and I can spend some time working on some of the 
subtasks.

 Heterogeneous Storage phase 2 - APIs to expose Storage Types
 

 Key: HDFS-5682
 URL: https://issues.apache.org/jira/browse/HDFS-5682
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Arpit Agarwal
Assignee: Arpit Agarwal

 Phase 1 (HDFS-2832) added support to present the DataNode as a collection of 
 discrete storages of different types.
 This Jira is to track phase 2 of the Heterogeneous Storage work which 
 involves exposing Storage Types to applications and adding Quota Management 
 support for administrators.
 This phase will also include tools support for administrators/users.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6373) Remove support for extended attributes on symlinks

Andrew Wang created HDFS-6373:
-

 Summary: Remove support for extended attributes on symlinks
 Key: HDFS-6373
 URL: https://issues.apache.org/jira/browse/HDFS-6373
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Andrew Wang


Looking in the Linux source code, we see the following:

http://lxr.linux.no/linux+v3.14.3/fs/xattr.c

{code}
  60/*
  61 * In the user.* namespace, only regular files and directories can 
have
  62 * extended attributes. For sticky directories, only the owner and
  63 * privileged users can write attributes.
  64 */
{code}

We should consider removing {{XAttrFeature}} from {{INodeSymlink}}.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6361) TestIdUserGroup.testUserUpdateSetting failed due to out of range nfsnobody Id

2014-05-12 Thread Yongjun Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-6361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995954#comment-13995954
 ] 

Yongjun Zhang commented on HDFS-6361:
-

Hi [~cmccabe] and [~brandonli],  

Thanks for the comments and info. I uploaded revised version 0002, in which I 
used Long to parse the string as Brandon suggested to make it general, and 
added a couple of more entries to the unit testcase. Thanks in advance for 
another round of review.




 TestIdUserGroup.testUserUpdateSetting failed due to out of range nfsnobody Id
 -

 Key: HDFS-6361
 URL: https://issues.apache.org/jira/browse/HDFS-6361
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: nfs
Affects Versions: 2.4.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
 Attachments: HDFS-6361.001.patch, HDFS-6361.002.patch


 The following error happens pretty often:
 org.apache.hadoop.nfs.nfs3.TestIdUserGroup.testUserUpdateSetting
 Failing for the past 1 build (Since Unstable#61 )
 Took 0.1 sec.
 add description
 Error Message
 For input string: 4294967294
 Stacktrace
 java.lang.NumberFormatException: For input string: 4294967294
   at 
 java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
   at java.lang.Integer.parseInt(Integer.java:495)
   at java.lang.Integer.valueOf(Integer.java:582)
   at 
 org.apache.hadoop.nfs.nfs3.IdUserGroup.updateMapInternal(IdUserGroup.java:137)
   at 
 org.apache.hadoop.nfs.nfs3.IdUserGroup.updateMaps(IdUserGroup.java:188)
   at org.apache.hadoop.nfs.nfs3.IdUserGroup.init(IdUserGroup.java:60)
   at 
 org.apache.hadoop.nfs.nfs3.TestIdUserGroup.testUserUpdateSetting(TestIdUserGroup.java:71)
 Standard Output
 log4j:WARN No appenders could be found for logger 
 (org.apache.hadoop.nfs.nfs3.IdUserGroup).
 log4j:WARN Please initialize the log4j system properly.
 log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
 info.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6325) Append should fail if the last block has insufficient number of replicas


[ 
https://issues.apache.org/jira/browse/HDFS-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995918#comment-13995918
 ] 

Keith Pak commented on HDFS-6325:
-

testFileAppend4 failure: 
There may be an overlap of shutdown of testAppendInsufficientLocations and 
startup of testCompleteOtherLeaseHoldersFile. 
I changed testAppendInsufficientLocations to use the field variable cluster 
instead of creating my own as a local variable and also reduced the number of 
DNs from 6 to 4. Otherwise, I see that testCompleteOtherLeaseHoldersFile is 
progressing so I think a bigger timeout may be required as stated in this JIRA: 
https://issues.apache.org/jira/browse/HADOOP-8596. 

TestBalancerWithNodeGroup: 
This also seems to be an intermittent issue from 
https://issues.apache.org/jira/browse/HDFS-6250. 







 Append should fail if the last block has insufficient number of replicas
 

 Key: HDFS-6325
 URL: https://issues.apache.org/jira/browse/HDFS-6325
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 2.2.0
Reporter: Konstantin Shvachko
Assignee: Keith Pak
 Attachments: HDFS-6325.patch, HDFS-6325.patch, HDFS-6325.patch, 
 HDFS-6325_test.patch, appendTest.patch


 Currently append() succeeds on a file with the last block that has no 
 replicas. But the subsequent updatePipeline() fails as there are no replicas 
 with the exception Unable to retrieve blocks locations for last block. This 
 leaves the file unclosed, and others can not do anything with it until its 
 lease expires.
 The solution is to check replicas of the last block on the NameNode and fail 
 during append() rather than during updatePipeline().
 How many replicas should be present before NN allows to append? I see two 
 options:
 # min-replication: allow append if the last block is minimally replicated (1 
 by default)
 # full-replication: allow append if the last block is fully replicated (3 by 
 default)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6325) Append should fail if the last block has insufficient number of replicas


 [ 
https://issues.apache.org/jira/browse/HDFS-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Keith Pak updated HDFS-6325:


Status: Patch Available  (was: In Progress)

 Append should fail if the last block has insufficient number of replicas
 

 Key: HDFS-6325
 URL: https://issues.apache.org/jira/browse/HDFS-6325
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 2.2.0
Reporter: Konstantin Shvachko
Assignee: Keith Pak
 Attachments: HDFS-6325.patch, HDFS-6325.patch, HDFS-6325_test.patch, 
 appendTest.patch


 Currently append() succeeds on a file with the last block that has no 
 replicas. But the subsequent updatePipeline() fails as there are no replicas 
 with the exception Unable to retrieve blocks locations for last block. This 
 leaves the file unclosed, and others can not do anything with it until its 
 lease expires.
 The solution is to check replicas of the last block on the NameNode and fail 
 during append() rather than during updatePipeline().
 How many replicas should be present before NN allows to append? I see two 
 options:
 # min-replication: allow append if the last block is minimally replicated (1 
 by default)
 # full-replication: allow append if the last block is fully replicated (3 by 
 default)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HDFS-6366) FsImage loading failed with RemoveXattr op


 [ 
https://issues.apache.org/jira/browse/HDFS-6366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Uma Maheswara Rao G resolved HDFS-6366.
---

   Resolution: Fixed
Fix Version/s: HDFS XAttrs (HDFS-2006)
 Hadoop Flags: Reviewed

Thanks for the review Yi. I have just committed this to branch!

 FsImage loading failed with RemoveXattr op
 --

 Key: HDFS-6366
 URL: https://issues.apache.org/jira/browse/HDFS-6366
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
 Fix For: HDFS XAttrs (HDFS-2006)

 Attachments: HDFS-6366.patch


 Should break after the removeXattr op loading.
 Otherwise fsimage loading will fail:
 java.io.IOException: Invalid operation read OP_REMOVE_XATTR
   at 
 org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:816)
   at 
 org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:227)
   at 
 org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:136)
   at 
 org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:805)
   at 
 org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:665)
   at 
 org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:272)
   at 
 org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:902)
   at 
 org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:651)
   at 
 org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:479)
   at 
 org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:535)
   at 
 org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:694)
   at 
 org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:679)
   at 
 org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1332)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6268) Better sorting in NetworkTopology#pseudoSortByDistance when no local node is found


[ 
https://issues.apache.org/jira/browse/HDFS-6268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995942#comment-13995942
 ] 

Andrew Wang commented on HDFS-6268:
---

Manually triggered Jenkins, dunno why it didn't run.

 Better sorting in NetworkTopology#pseudoSortByDistance when no local node is 
 found
 --

 Key: HDFS-6268
 URL: https://issues.apache.org/jira/browse/HDFS-6268
 Project: Hadoop HDFS
  Issue Type: Improvement
Affects Versions: 2.4.0
Reporter: Andrew Wang
Assignee: Andrew Wang
Priority: Minor
 Attachments: hdfs-6268-1.patch, hdfs-6268-2.patch, hdfs-6268-3.patch, 
 hdfs-6268-4.patch


 In NetworkTopology#pseudoSortByDistance, if no local node is found, it will 
 always place the first rack local node in the list in front.
 This became an issue when a dataset was loaded from a single datanode. This 
 datanode ended up being the first replica for all the blocks in the dataset. 
 When running an Impala query, the non-local reads when reading past a block 
 boundary were all hitting this node, meaning massive load skew.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Work started] (HDFS-6372) Handle setXattr rpcIDs for OfflineEditsViewer


 [ 
https://issues.apache.org/jira/browse/HDFS-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HDFS-6372 started by Uma Maheswara Rao G.

 Handle setXattr rpcIDs for OfflineEditsViewer
 -

 Key: HDFS-6372
 URL: https://issues.apache.org/jira/browse/HDFS-6372
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Affects Versions: HDFS XAttrs (HDFS-2006)
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
 Attachments: HDFS-6372.patch, editsStored


 toXml and fromXml methods in SetXattrOp should store and read the rpcids.
 This JIRA can also fix the failures of 
 TestOfflineEditsViewer
 TestRetryCacheWithHA 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6359) WebHdfs NN servlet issues redirects in safemode or standby

Daryn Sharp created HDFS-6359:
-

 Summary: WebHdfs NN servlet issues redirects in safemode or standby
 Key: HDFS-6359
 URL: https://issues.apache.org/jira/browse/HDFS-6359
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Priority: Critical


Webhdfs does not check for safemode or standby during issuing a redirect for 
open/create/checksum calls.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6377) Unify xattr name and value limits into a single limit


 [ 
https://issues.apache.org/jira/browse/HDFS-6377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang updated HDFS-6377:
--

Attachment: hdfs-6377-1.patch

Patch attached unifying these two. I also cleaned up the xattr tests a bit. Ran 
TestFileContextXAttr and TestNameNodeXAttr successfully.

 Unify xattr name and value limits into a single limit
 -

 Key: HDFS-6377
 URL: https://issues.apache.org/jira/browse/HDFS-6377
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Affects Versions: HDFS XAttrs (HDFS-2006)
Reporter: Andrew Wang
Assignee: Andrew Wang
 Attachments: hdfs-6377-1.patch


 Instead of having separate limits and config options for the size of an 
 xattr's name and value, let's use a single limit.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Work started] (HDFS-6377) Unify xattr name and value limits into a single limit


 [ 
https://issues.apache.org/jira/browse/HDFS-6377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HDFS-6377 started by Andrew Wang.

 Unify xattr name and value limits into a single limit
 -

 Key: HDFS-6377
 URL: https://issues.apache.org/jira/browse/HDFS-6377
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Affects Versions: HDFS XAttrs (HDFS-2006)
Reporter: Andrew Wang
Assignee: Andrew Wang
 Attachments: hdfs-6377-1.patch


 Instead of having separate limits and config options for the size of an 
 xattr's name and value, let's use a single limit.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6186) Pause deletion of blocks when the namenode starts up

2014-05-12 Thread Jing Zhao (JIRA)

[
https://issues.apache.org/jira/browse/HDFS-6186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jing Zhao updated HDFS-6186:

Attachment: HDFS-6186.003.patch

Thanks for the review, Suresh! Update the patch to address your comments.

Pause deletion of blocks when the namenode starts up

Key: HDFS-6186
URL: https://issues.apache.org/jira/browse/HDFS-6186
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Reporter: Suresh Srinivas
Assignee: Jing Zhao
Attachments: HDFS-6186.000.patch, HDFS-6186.002.patch,
HDFS-6186.003.patch

HDFS namenode can delete blocks very quickly, given the deletion happens as a
parallel operation spread across many datanodes. One of the frequent
anxieties I see is that a lot of data can be deleted very quickly, when a
cluster is brought up, especially when one of the storage directories has
failed and namenode metadata was copied from another storage. Copying wrong
metadata would results in some of the newer files (if old metadata was
copied) being deleted along with their blocks.
HDFS-5986 now captures the number of pending deletion block on namenode webUI
and JMX. I propose pausing deletion of blocks for a configured period of time
(default 1 hour?) after namenode comes out of safemode. This will give enough
time for the administrator to notice large number of pending deletion blocks
and take corrective action.
Thoughts?

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6056) Clean up NFS config settings

2014-05-12 Thread Brandon Li (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brandon Li updated HDFS-6056:
-

Hadoop Flags: Incompatible change

 Clean up NFS config settings
 

 Key: HDFS-6056
 URL: https://issues.apache.org/jira/browse/HDFS-6056
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: nfs
Affects Versions: 2.3.0
Reporter: Aaron T. Myers
Assignee: Brandon Li
 Attachments: HDFS-6056.001.patch, HDFS-6056.002.patch, 
 HDFS-6056.003.patch


 As discussed on HDFS-6050, there's a few opportunities to improve the config 
 settings related to NFS. This JIRA is to implement those changes.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-5522) Datanode disk error check may be incorrectly skipped

2014-05-12 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-5522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995980#comment-13995980
 ] 

Hudson commented on HDFS-5522:
--

FAILURE: Integrated in Hadoop-trunk-Commit #5604 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/5604/])
HDFS-5522. Datanode disk error check may be incorrectly skipped. Contributed by 
Rushabh Shah. (kihwal: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1594055)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDiskError.java


 Datanode disk error check may be incorrectly skipped
 

 Key: HDFS-5522
 URL: https://issues.apache.org/jira/browse/HDFS-5522
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 0.23.9, 2.2.0
Reporter: Kihwal Lee
Assignee: Rushabh S Shah
 Fix For: 3.0.0, 2.5.0

 Attachments: HDFS-5522-v2.patch, HDFS-5522-v3.patch, HDFS-5522.patch


 After HDFS-4581 and HDFS-4699, {{checkDiskError()}} is not called when 
 network errors occur during processing data node requests.  This appears to 
 create problems when a disk is having problems, but not failing I/O soon. 
 If I/O hangs for a long time, network read/write may timeout first and the 
 peer may close the connection. Although the error was caused by a faulty 
 local disk, disk check is not being carried out in this case. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6376) Distcp data between two HA clusters requires another configuration

2014-05-12 Thread Dave Marion (JIRA)

Dave Marion created HDFS-6376:
-

 Summary: Distcp data between two HA clusters requires another 
configuration
 Key: HDFS-6376
 URL: https://issues.apache.org/jira/browse/HDFS-6376
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: federation
Affects Versions: 2.3.0
 Environment: CDH 5.0 (Apache 2.3.0 + some patches)
Reporter: Dave Marion


User has to create a third set of configuration files for distcp when 
transferring data between two HA clusters.

Consider the scenario in [1]. You cannot put all of the required properties in 
core-site.xml and hdfs-site.xml for the client to resolve the location of both 
active namenodes. If you do, then the datanodes from cluster A may join cluster 
B. I can not find a configuration option that tells the datanodes to federate 
blocks for only one of the clusters in the configuration.


[1] 
http://mail-archives.apache.org/mod_mbox/hadoop-user/201404.mbox/%3CBAY172-W2133964E0C283968C161DD1520%40phx.gbl%3E



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6334) Client failover proxy provider for IP failover based NN HA

2014-05-12 Thread Aaron T. Myers (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-6334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995957#comment-13995957
 ] 

Aaron T. Myers commented on HDFS-6334:
--

Patch looks great to me, Kihwal. Thanks for doing it. Two little nits:

# bq. +   * used, a special token handling mat be needed to make sure a token 
acquired 
s/mat/may/g
# The method comment for {{ConfiguredFailoverProxyProvider#useLogicalURI}} says 
Logical URI is not used for IP failover. While strictly true, I'm guessing 
you meant to put a different comment here.

+1 once these are addressed.

 Client failover proxy provider for IP failover based NN HA
 --

 Key: HDFS-6334
 URL: https://issues.apache.org/jira/browse/HDFS-6334
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Kihwal Lee
Assignee: Kihwal Lee
 Attachments: HDFS-6334.patch, HDFS-6334.v2.patch


 With RPCv9 and improvements in the SPNEGO auth handling, it is possible to 
 set up a pair of HA namenodes utilizing IP failover as client-request fencing 
 mechanism.
 This jira will make it possible for HA to be configured without requiring use 
 of logical URI and provide a simple IP failover proxy provider.  The change 
 will allow any old implementation of {{FailoverProxyProvider}} to continue to 
 work.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6371) In HA setup, the standby NN should redirect WebHDFS write requests to the active NN

2014-05-12 Thread Tsz Wo Nicholas Sze (JIRA)

[
https://issues.apache.org/jira/browse/HDFS-6371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Tsz Wo Nicholas Sze updated HDFS-6371:
--

Component/s: namenode
Issue Type: Improvement (was: Bug)

In HA setup, the standby NN should redirect WebHDFS write requests to the
active NN
---

Key: HDFS-6371
URL: https://issues.apache.org/jira/browse/HDFS-6371
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode, webhdfs
Reporter: Tsz Wo Nicholas Sze
Assignee: Tsz Wo Nicholas Sze

The current WebHDFS implementation in namenode does not check its HA state --
it does the same thing no matter it is active or standby.
Suppose a http client talk to the standby NN via WebHDFS. For the read
operations, there is no problem. For the write operations, if the operation
requires http redirect (e.g. creating a file), it will work since the standby
NN will also redirect the client to a DN. When the client connect to the DN,
the DN will fulfill the request with the active NN. However, for the write
operations not requiring http redirect (e.g. mkdir), the operation will fail
with StandbyException since it will be executed with the standby NN.
There are two solutions:
# The http client could catch StandbyException and then retries with the
other NN in this case.
# The standby NN redirects the request to the active NN.
The second solution seems better since the client does not need to know both
active NN and standby NN.
Note that WebHdfsFileSystem is already able to handle HA failover. The JIRA
is for other http clients.

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6186) Pause deletion of blocks when the namenode starts up


[ 
https://issues.apache.org/jira/browse/HDFS-6186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996006#comment-13996006
 ] 

Hadoop QA commented on HDFS-6186:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12644511/HDFS-6186.003.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-hdfs-project/hadoop-hdfs:

  org.apache.hadoop.hdfs.server.namenode.TestCheckpoint

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HDFS-Build/6884//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6884//console

This message is automatically generated.

 Pause deletion of blocks when the namenode starts up
 

 Key: HDFS-6186
 URL: https://issues.apache.org/jira/browse/HDFS-6186
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Reporter: Suresh Srinivas
Assignee: Jing Zhao
 Attachments: HDFS-6186.000.patch, HDFS-6186.002.patch, 
 HDFS-6186.003.patch


 HDFS namenode can delete blocks very quickly, given the deletion happens as a 
 parallel operation spread across many datanodes. One of the frequent 
 anxieties I see is that a lot of data can be deleted very quickly, when a 
 cluster is brought up, especially when one of the storage directories has 
 failed and namenode metadata was copied from another storage. Copying wrong 
 metadata would results in some of the newer files (if old metadata was 
 copied) being deleted along with their blocks. 
 HDFS-5986 now captures the number of pending deletion block on namenode webUI 
 and JMX. I propose pausing deletion of blocks for a configured period of time 
 (default 1 hour?) after namenode comes out of safemode. This will give enough 
 time for the administrator to notice large number of pending deletion blocks 
 and take corrective action.
 Thoughts?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-5682) Heterogeneous Storage phase 2 - APIs to expose Storage Types

2014-05-12 Thread Nick Dimiduk (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-5682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996033#comment-13996033
 ] 

Nick Dimiduk commented on HDFS-5682:


Hi [~wuzesheng]. Can you describe your SLA requirements in detail? Maybe these 
needs can be addressed in other ways?

 Heterogeneous Storage phase 2 - APIs to expose Storage Types
 

 Key: HDFS-5682
 URL: https://issues.apache.org/jira/browse/HDFS-5682
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Arpit Agarwal
Assignee: Arpit Agarwal

 Phase 1 (HDFS-2832) added support to present the DataNode as a collection of 
 discrete storages of different types.
 This Jira is to track phase 2 of the Heterogeneous Storage work which 
 involves exposing Storage Types to applications and adding Quota Management 
 support for administrators.
 This phase will also include tools support for administrators/users.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HDFS-6372) Handle setXattr rpcIDs for OfflineEditsViewer


 [ 
https://issues.apache.org/jira/browse/HDFS-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Uma Maheswara Rao G resolved HDFS-6372.
---

   Resolution: Fixed
Fix Version/s: HDFS XAttrs (HDFS-2006)
 Hadoop Flags: Reviewed

 Handle setXattr rpcIDs for OfflineEditsViewer
 -

 Key: HDFS-6372
 URL: https://issues.apache.org/jira/browse/HDFS-6372
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Affects Versions: HDFS XAttrs (HDFS-2006)
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
 Fix For: HDFS XAttrs (HDFS-2006)

 Attachments: HDFS-6372.patch, editsStored


 toXml and fromXml methods in SetXattrOp should store and read the rpcids.
 This JIRA can also fix the failures of 
 TestOfflineEditsViewer
 TestRetryCacheWithHA 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6283) Write end user documentation for xattrs.


[ 
https://issues.apache.org/jira/browse/HDFS-6283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995978#comment-13995978
 ] 

Yi Liu commented on HDFS-6283:
--

Thanks Andrew. Sure, you can pick this one up:)

 Write end user documentation for xattrs.
 

 Key: HDFS-6283
 URL: https://issues.apache.org/jira/browse/HDFS-6283
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: documentation
Reporter: Chris Nauroth
Assignee: Andrew Wang
 Attachments: hdfs-6283-1.patch


 Update the File System Shell documentation to cover the new getfattr and 
 setfattr commands.  If warranted, consider adding a separate dedicated page 
 for fuller discussion of the xattrs model and how the feature works in more 
 detail.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-5682) Heterogeneous Storage phase 2 - APIs to expose Storage Types

2014-05-12 Thread Arpit Agarwal (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-5682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995299#comment-13995299
 ] 

Arpit Agarwal commented on HDFS-5682:
-

Hi [~wuzesheng], this was deprioritized due to 2.4 stabilization.

We will post a document describing the proposed API within the next couple of 
weeks, followed shortly by the detailed design doc.

 Heterogeneous Storage phase 2 - APIs to expose Storage Types
 

 Key: HDFS-5682
 URL: https://issues.apache.org/jira/browse/HDFS-5682
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Arpit Agarwal
Assignee: Arpit Agarwal

 Phase 1 (HDFS-2832) added support to present the DataNode as a collection of 
 discrete storages of different types.
 This Jira is to track phase 2 of the Heterogeneous Storage work which 
 involves exposing Storage Types to applications and adding Quota Management 
 support for administrators.
 This phase will also include tools support for administrators/users.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6373) Remove support for extended attributes on symlinks


[ 
https://issues.apache.org/jira/browse/HDFS-6373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995982#comment-13995982
 ] 

Yi Liu commented on HDFS-6373:
--

Thanks Andrew.
Let's remove XAttrFeature from INodeSysmlink, so symlinks don't have xattrs of 
their own.

In design doc we have 
{quote}
Symlinks do not have XAttrs of their own. Operations that modify the XAttrs of 
a sysmlink
instead modify the XAttr of the symlink’s target
{quote}

 Remove support for extended attributes on symlinks
 --

 Key: HDFS-6373
 URL: https://issues.apache.org/jira/browse/HDFS-6373
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Reporter: Andrew Wang
Assignee: Charles Lamb

 Looking in the Linux source code, we see the following:
 http://lxr.linux.no/linux+v3.14.3/fs/xattr.c
 {code}
   60/*
   61 * In the user.* namespace, only regular files and directories 
 can have
   62 * extended attributes. For sticky directories, only the owner and
   63 * privileged users can write attributes.
   64 */
 {code}
 We should consider removing {{XAttrFeature}} from {{INodeSymlink}}.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HDFS-6325) Append should fail if the last block has insufficient number of replicas


 [ 
https://issues.apache.org/jira/browse/HDFS-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Keith Pak updated HDFS-6325:


Attachment: HDFS-6325.patch

 Append should fail if the last block has insufficient number of replicas
 

 Key: HDFS-6325
 URL: https://issues.apache.org/jira/browse/HDFS-6325
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 2.2.0
Reporter: Konstantin Shvachko
Assignee: Keith Pak
 Attachments: HDFS-6325.patch, HDFS-6325.patch, HDFS-6325.patch, 
 HDFS-6325_test.patch, appendTest.patch


 Currently append() succeeds on a file with the last block that has no 
 replicas. But the subsequent updatePipeline() fails as there are no replicas 
 with the exception Unable to retrieve blocks locations for last block. This 
 leaves the file unclosed, and others can not do anything with it until its 
 lease expires.
 The solution is to check replicas of the last block on the NameNode and fail 
 during append() rather than during updatePipeline().
 How many replicas should be present before NN allows to append? I see two 
 options:
 # min-replication: allow append if the last block is minimally replicated (1 
 by default)
 # full-replication: allow append if the last block is fully replicated (3 by 
 default)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6361) TestIdUserGroup.testUserUpdateSetting failed due to out of range nfsnobody Id


[ 
https://issues.apache.org/jira/browse/HDFS-6361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996031#comment-13996031
 ] 

Hadoop QA commented on HDFS-6361:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12644540/HDFS-6361.002.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:red}-1 findbugs{color}.  The patch appears to introduce 1 new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-common-project/hadoop-nfs.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HDFS-Build/6887//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HDFS-Build/6887//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-nfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6887//console

This message is automatically generated.

 TestIdUserGroup.testUserUpdateSetting failed due to out of range nfsnobody Id
 -

 Key: HDFS-6361
 URL: https://issues.apache.org/jira/browse/HDFS-6361
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: nfs
Affects Versions: 2.4.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
 Attachments: HDFS-6361.001.patch, HDFS-6361.002.patch


 The following error happens pretty often:
 org.apache.hadoop.nfs.nfs3.TestIdUserGroup.testUserUpdateSetting
 Failing for the past 1 build (Since Unstable#61 )
 Took 0.1 sec.
 add description
 Error Message
 For input string: 4294967294
 Stacktrace
 java.lang.NumberFormatException: For input string: 4294967294
   at 
 java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
   at java.lang.Integer.parseInt(Integer.java:495)
   at java.lang.Integer.valueOf(Integer.java:582)
   at 
 org.apache.hadoop.nfs.nfs3.IdUserGroup.updateMapInternal(IdUserGroup.java:137)
   at 
 org.apache.hadoop.nfs.nfs3.IdUserGroup.updateMaps(IdUserGroup.java:188)
   at org.apache.hadoop.nfs.nfs3.IdUserGroup.init(IdUserGroup.java:60)
   at 
 org.apache.hadoop.nfs.nfs3.TestIdUserGroup.testUserUpdateSetting(TestIdUserGroup.java:71)
 Standard Output
 log4j:WARN No appenders could be found for logger 
 (org.apache.hadoop.nfs.nfs3.IdUserGroup).
 log4j:WARN Please initialize the log4j system properly.
 log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
 info.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6344) Maximum limit on the size of an xattr


[ 
https://issues.apache.org/jira/browse/HDFS-6344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995968#comment-13995968
 ] 

Yi Liu commented on HDFS-6344:
--

Thanks Chris. Make sense, consistency is important. Let's address it in 
HDFS-6377

 Maximum limit on the size of an xattr
 -

 Key: HDFS-6344
 URL: https://issues.apache.org/jira/browse/HDFS-6344
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Affects Versions: HDFS XAttrs (HDFS-2006)
Reporter: Andrew Wang
Assignee: Yi Liu
 Fix For: HDFS XAttrs (HDFS-2006)

 Attachments: HDFS-6344.patch


 We should support limits on the maximum size of an xattr name/value.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HDFS-4437) Clients should not retain leases on moved files