[jira] [Commented] (HDFS-6283) Write end user documentation for xattrs.
[
https://issues.apache.org/jira/browse/HDFS-6283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997300#comment-13997300
]
Uma Maheswara Rao G commented on HDFS-6283:
---
{quote}
extended attributes are not interpreted by the system and are instead used by
applications to store additional information about an inode.
{quote}
I saw this line. This is saying Xattrs can not be interpreted by System. But
the xattrs with system/security name will be used only by system. So, we should
not generically say that Xattrs can not be interpreted by system? Please
correct me I interpreted this line wrongly :-)
Let me know if we need to change this, then I wil file a follow-up JIRA as it
is already committed.
Write end user documentation for xattrs.
Key: HDFS-6283
URL: https://issues.apache.org/jira/browse/HDFS-6283
Project: Hadoop HDFS
Issue Type: Sub-task
Components: documentation
Reporter: Chris Nauroth
Assignee: Andrew Wang
Fix For: HDFS XAttrs (HDFS-2006)
Attachments: hdfs-6283-1.patch, hdfs-6283-2.patch, hdfs-6283-3.patch
Update the File System Shell documentation to cover the new getfattr and
setfattr commands. If warranted, consider adding a separate dedicated page
for fuller discussion of the xattrs model and how the feature works in more
detail.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6381) Fix a typo in INodeReference.java
[
https://issues.apache.org/jira/browse/HDFS-6381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-6381:
Resolution: Fixed
Fix Version/s: 2.5.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I've committed this to trunk and branch-2. Thanks for the fix [~decster]!
Fix a typo in INodeReference.java
-
Key: HDFS-6381
URL: https://issues.apache.org/jira/browse/HDFS-6381
Project: Hadoop HDFS
Issue Type: Bug
Components: documentation
Reporter: Binglin Chang
Assignee: Binglin Chang
Priority: Trivial
Fix For: 2.5.0
Attachments: HDFS-6381.v1.patch
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java
{code}
* For example,
- * (1) Support we have /abc/foo, say the inode of foo is
inode(id=1000,name=foo)
+ * (1) Suppose we have /abc/foo, say the inode of foo is
inode(id=1000,name=foo)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6346) Optimize OP_SET_XATTRS by persisting single Xattr entry per setXattr/removeXattr api call
[
https://issues.apache.org/jira/browse/HDFS-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liu updated HDFS-6346:
-
Attachment: HDFS-6346.1.patch
Thanks Uma for the review.
{quote}
Do you think, we need warn/log to the user that xattr attempting to remove does
not exist?
{quote}
make sense, let's add a log when removing nonexistent xattr.
{quote}
Pls remove empty line between
{quote}
OK
{quote}
When existing and updated xattrs equal then we need not even call
updateINodeXAttr. So code can be changed to
{code}
if (existingXAttrs.size() != newXAttrs.size()) {
XAttrStorage.updateINodeXAttrs(inode, newXAttrs, snapshotId);
return xAttr;
}
return null;
{code}
{quote}
Right, I will update it
Optimize OP_SET_XATTRS by persisting single Xattr entry per
setXattr/removeXattr api call
-
Key: HDFS-6346
URL: https://issues.apache.org/jira/browse/HDFS-6346
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Reporter: Uma Maheswara Rao G
Assignee: Yi Liu
Attachments: HDFS-6346.1.patch, HDFS-6346.2.patch, HDFS-6346.patch,
editsStored
When a new xattrs set on an Inode, it may add this with OP_SET_XATTRS and
let's say [user.name1:value1]
On a next call if we set another xattrs, then it may store along with older
existing xattrs again. It may be like [user.name1:value1, user.name2:value2]
So, on adding more xattrs on same Inode, that list may grow and we keep store
the entries of already existing name, value fairs.
Right now we defaulted the max Xattrs on an Inode to 32 and configured. If
user modified it to much larger value and start setting xattrs, then edits
loading may create issue like my above example.
But I didn't refer any usecase of having large number of xattrs, this is just
the assumption to consider a case. My biggest doubt is whether we will have
such real usecases to have huge xattrs on a single INode.
So, here is a thought on having OP_SET_XATTR for each setXAttr operation to
be logged, When we replay them we need to consolidate. This is some initial
thought we can think more if others also feel we need to consider this case
to handle.
Otherwise we endup storing Xattrs entries in editlog file as n(n+1)/2 where n
is number xattrs for a file/dir. This may be issue only when we have large
number configured max xattrs for inode.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6376) Distcp data between two HA clusters requires another configuration
[ https://issues.apache.org/jira/browse/HDFS-6376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Marion updated HDFS-6376: -- Attachment: HDFS-6376-2.patch Patch 2 against the 2.4 branch Distcp data between two HA clusters requires another configuration -- Key: HDFS-6376 URL: https://issues.apache.org/jira/browse/HDFS-6376 Project: Hadoop HDFS Issue Type: Bug Components: federation Affects Versions: 2.3.0, 2.4.0 Environment: CDH 5.0 (Apache 2.3.0 + some patches) Reporter: Dave Marion Attachments: HDFS-6376-2.patch, HDFS-6376-patch-1.patch User has to create a third set of configuration files for distcp when transferring data between two HA clusters. Consider the scenario in [1]. You cannot put all of the required properties in core-site.xml and hdfs-site.xml for the client to resolve the location of both active namenodes. If you do, then the datanodes from cluster A may join cluster B. I can not find a configuration option that tells the datanodes to federate blocks for only one of the clusters in the configuration. [1] http://mail-archives.apache.org/mod_mbox/hadoop-user/201404.mbox/%3CBAY172-W2133964E0C283968C161DD1520%40phx.gbl%3E -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-2006) ability to support storing extended attributes per file
[ https://issues.apache.org/jira/browse/HDFS-2006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-2006: -- Attachment: ExtendedAttributes.html Attached the doc(html) generated for Extended Attributes for reference ability to support storing extended attributes per file --- Key: HDFS-2006 URL: https://issues.apache.org/jira/browse/HDFS-2006 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: HDFS XAttrs (HDFS-2006) Reporter: dhruba borthakur Assignee: Yi Liu Attachments: ExtendedAttributes.html, HDFS-XAttrs-Design-1.pdf, HDFS-XAttrs-Design-2.pdf, HDFS-XAttrs-Design-3.pdf, Test-Plan-for-Extended-Attributes-1.pdf, xattrs.1.patch, xattrs.patch It would be nice if HDFS provides a feature to store extended attributes for files, similar to the one described here: http://en.wikipedia.org/wiki/Extended_file_attributes. The challenge is that it has to be done in such a way that a site not using this feature does not waste precious memory resources in the namenode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6293) Issues with OIV processing PB-based fsimages
[ https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6293: - Attachment: HDFS-6293.v2.trunk.patch HDFS-6293.v2.branch-2.patch Fixed warnings in the new set of patches. Two new deprecation warnings remain. They should go away after HDFS-6384. Issues with OIV processing PB-based fsimages Key: HDFS-6293 URL: https://issues.apache.org/jira/browse/HDFS-6293 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.4.0 Reporter: Kihwal Lee Assignee: Kihwal Lee Priority: Blocker Attachments: HDFS-6293.000.patch, HDFS-6293.001.patch, HDFS-6293.002-save-deprecated-fsimage.patch, HDFS-6293.branch-2.patch, HDFS-6293.trunk.patch, HDFS-6293.trunk.patch, HDFS-6293.v2.branch-2.patch, HDFS-6293.v2.trunk.patch, HDFS-6293_sbn_ckpt_retention.patch, HDFS-6293_sbn_ckpt_retention_oiv_legacy.patch, Heap Histogram.html There are issues with OIV when processing fsimages in protobuf. Due to the internal layout changes introduced by the protobuf-based fsimage, OIV consumes excessive amount of memory. We have tested with a fsimage with about 140M files/directories. The peak heap usage when processing this image in pre-protobuf (i.e. pre-2.4.0) format was about 350MB. After converting the image to the protobuf format on 2.4.0, OIV would OOM even with 80GB of heap (max new size was 1GB). It should be possible to process any image with the default heap size of 1.5GB. Another issue is the complete change of format/content in OIV's XML output. I also noticed that the secret manager section has no tokens while there were unexpired tokens in the original image (pre-2.4.0). I did not check whether they were also missing in the new pb fsimage. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6383) Upgrade S3n s3.fs.buffer.dir to suppoer multi directories
[ https://issues.apache.org/jira/browse/HDFS-6383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-6383: - Target Version/s: 2.5.0 Affects Version/s: 2.4.0 Assignee: Ted Malaska Thanks a lot for the patch, Ted. I've assigned it to you and will go see if I can figure out why the pre-commit Jenkins job hasn't run yet. Upgrade S3n s3.fs.buffer.dir to suppoer multi directories - Key: HDFS-6383 URL: https://issues.apache.org/jira/browse/HDFS-6383 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 2.4.0 Reporter: Ted Malaska Assignee: Ted Malaska Priority: Minor Attachments: HDFS-6383.patch s3.fs.buffer.dir defines the tmp folder where files will be written to before getting sent to S3. Right now this is limited to a single folder which causes to major issues. 1. You need a drive with enough space to store all the tmp files at once 2. You are limited to the IO speeds of a single drive This solution will resolve both and has been tested to increase the S3 write speed by 2.5x with 10 mappers on hs1. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6377) Unify xattr name and value limits into a single limit
[ https://issues.apache.org/jira/browse/HDFS-6377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996988#comment-13996988 ] Andrew Wang commented on HDFS-6377: --- Hey Chris, good comments. Let's remove the import in HDFS-6395. The cacheManager check was because of the new unit test I added for testing the Precondition checks. Basically, if the NN fails in its constructor, it calls stopActiveServices to clean up. Since FSDirectory can throw a precondition, this means we try to clean up while cacheManager is still null. This meant I was getting a NullPointerException rather than the IllegalArgumentException I wanted to verify, and I couldn't get the LogVerificationAppender to show me the stack trace. Thus, the null check :) Unify xattr name and value limits into a single limit - Key: HDFS-6377 URL: https://issues.apache.org/jira/browse/HDFS-6377 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: HDFS XAttrs (HDFS-2006) Reporter: Andrew Wang Assignee: Andrew Wang Fix For: HDFS XAttrs (HDFS-2006) Attachments: hdfs-6377-1.patch, hdfs-6377-2.patch Instead of having separate limits and config options for the size of an xattr's name and value, let's use a single limit. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6389) Rename restrictions for encryption zones
Alejandro Abdelnur created HDFS-6389: Summary: Rename restrictions for encryption zones Key: HDFS-6389 URL: https://issues.apache.org/jira/browse/HDFS-6389 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode, security Reporter: Alejandro Abdelnur Assignee: Charles Lamb Files and directories should not be moved in or out an encryption zone. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6250) TestBalancerWithNodeGroup.testBalancerWithRackLocality fails
[
https://issues.apache.org/jira/browse/HDFS-6250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997229#comment-13997229
]
Binglin Chang commented on HDFS-6250:
-
Hi Junping,
run TestBalancerWithNodeGroup.testBalancerWithRackLocality 50 times, no failure
or timeout, average running time 5.2s, before was 20s
run TestBalancerWithNodeGroup.testBalancerWithNodeGroup 50 times, no failure or
timeout, average running time 10.2s, before was 17s
TestBalancerWithNodeGroup.testBalancerWithRackLocality fails
Key: HDFS-6250
URL: https://issues.apache.org/jira/browse/HDFS-6250
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Kihwal Lee
Assignee: Chen He
Attachments: HDFS-6250-v2.patch, HDFS-6250-v3.patch, HDFS-6250.patch,
test_log.txt
It was seen in https://builds.apache.org/job/PreCommit-HDFS-Build/6669/
{panel}
java.lang.AssertionError: expected:1800 but was:1810
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:147)
at org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
.testBalancerWithRackLocality(TestBalancerWithNodeGroup.java:253)
{panel}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-4258) Rename of Being Written Files
[ https://issues.apache.org/jira/browse/HDFS-4258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-4258: --- Resolution: Duplicate Status: Resolved (was: Patch Available) I think this JIRA / patch has gotten a bit stale, since INode IDs are already in, as well as HDFS-6294, which addressed the issues we had with moving files that were open for write. I'm marking this as a dupe of HDFS-6294. Feel free to reopen if there's something more we should do here. Rename of Being Written Files - Key: HDFS-4258 URL: https://issues.apache.org/jira/browse/HDFS-4258 Project: Hadoop HDFS Issue Type: Bug Components: hdfs-client, namenode Affects Versions: 3.0.0 Reporter: Tsz Wo Nicholas Sze Assignee: Brandon Li Attachments: HDFS-4258.patch, HDFS-4258.patch, HDFS-4258.patch, HDFS-4258.patch When a being written file or it's ancestor directories is renamed, the path in the file lease is also renamed. Then the writer of the file usually will fail since the file path in the writer is not updated. Moreover, I think there is a bug as follow: # Client writes 0's to F_0=/foo/file and writes 1's to F_1=/bar/file at the same time. # Rename /bar to /baz # Rename /foo to /bar Then, writing to F_0 will fail since /foo/file does not exist anymore but writing to F_1 may succeed since /bar/file exits as a different file. In such case, the content of /bar/file could be partly 0's and partly 1's. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6186) Pause deletion of blocks when the namenode starts up
[ https://issues.apache.org/jira/browse/HDFS-6186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-6186: Resolution: Fixed Fix Version/s: 2.5.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for the review again, Suresh! I've committed this to trunk and branch-2. I've also created HDFS-6385 for showing the pending deletion information in Web UI. Pause deletion of blocks when the namenode starts up Key: HDFS-6186 URL: https://issues.apache.org/jira/browse/HDFS-6186 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Suresh Srinivas Assignee: Jing Zhao Fix For: 2.5.0 Attachments: HDFS-6186.000.patch, HDFS-6186.002.patch, HDFS-6186.003.patch HDFS namenode can delete blocks very quickly, given the deletion happens as a parallel operation spread across many datanodes. One of the frequent anxieties I see is that a lot of data can be deleted very quickly, when a cluster is brought up, especially when one of the storage directories has failed and namenode metadata was copied from another storage. Copying wrong metadata would results in some of the newer files (if old metadata was copied) being deleted along with their blocks. HDFS-5986 now captures the number of pending deletion block on namenode webUI and JMX. I propose pausing deletion of blocks for a configured period of time (default 1 hour?) after namenode comes out of safemode. This will give enough time for the administrator to notice large number of pending deletion blocks and take corrective action. Thoughts? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6334) Client failover proxy provider for IP failover based NN HA
[
https://issues.apache.org/jira/browse/HDFS-6334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-6334:
-
Attachment: HDFS-6334.v3.patch
This is the diff between v2 and v3 of the patch. All changes are in comment. So
I won't wait for precommit.
{panel}
293c293
+ * used, a special token handling mat be needed to make sure a token
acquired
---
+ * used, a special token handling may be needed to make sure a token
acquired
338c338
+ * Logical URI is not used for IP failover.
---
+ * Logical URI is required for this failover proxy provider.
{panel}
Client failover proxy provider for IP failover based NN HA
--
Key: HDFS-6334
URL: https://issues.apache.org/jira/browse/HDFS-6334
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Attachments: HDFS-6334.patch, HDFS-6334.v2.patch, HDFS-6334.v3.patch
With RPCv9 and improvements in the SPNEGO auth handling, it is possible to
set up a pair of HA namenodes utilizing IP failover as client-request fencing
mechanism.
This jira will make it possible for HA to be configured without requiring use
of logical URI and provide a simple IP failover proxy provider. The change
will allow any old implementation of {{FailoverProxyProvider}} to continue to
work.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6325) Append should fail if the last block has insufficient number of replicas
[ https://issues.apache.org/jira/browse/HDFS-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keith Pak updated HDFS-6325: Attachment: (was: HDFS-6325.patch) Append should fail if the last block has insufficient number of replicas Key: HDFS-6325 URL: https://issues.apache.org/jira/browse/HDFS-6325 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.2.0 Reporter: Konstantin Shvachko Assignee: Keith Pak Attachments: HDFS-6325.patch, HDFS-6325_test.patch, appendTest.patch Currently append() succeeds on a file with the last block that has no replicas. But the subsequent updatePipeline() fails as there are no replicas with the exception Unable to retrieve blocks locations for last block. This leaves the file unclosed, and others can not do anything with it until its lease expires. The solution is to check replicas of the last block on the NameNode and fail during append() rather than during updatePipeline(). How many replicas should be present before NN allows to append? I see two options: # min-replication: allow append if the last block is minimally replicated (1 by default) # full-replication: allow append if the last block is fully replicated (3 by default) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6193) HftpFileSystem open should throw FileNotFoundException for non-existing paths
[ https://issues.apache.org/jira/browse/HDFS-6193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13992986#comment-13992986 ] Haohui Mai commented on HDFS-6193: -- I don't think this is a blocker since hftp / hsftp have been deprecated and been superseded by webhdfs. It looks to me that the performance impact is still up to debate (the same fix has been applied to webhdfs in HDFS-6143, see the comments for the details). I'm moving it out to unblock 2.4.1. Feel free to move it back you think it is essential for the release. HftpFileSystem open should throw FileNotFoundException for non-existing paths - Key: HDFS-6193 URL: https://issues.apache.org/jira/browse/HDFS-6193 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.4.0 Reporter: Gera Shegalov Assignee: Gera Shegalov Priority: Blocker Attachments: HDFS-6193-branch-2.4.0.v01.patch, HDFS-6193-branch-2.4.v02.patch WebHdfsFileSystem.open and HftpFileSystem.open incorrectly handles non-existing paths. - 'open', does not really open anything, i.e., it does not contact the server, and therefore cannot discover FileNotFound, it's deferred until next read. It's counterintuitive and not how local FS or HDFS work. In POSIX you get ENOENT on open. [LzoInputFormat.getSplits|https://github.com/kevinweil/elephant-bird/blob/master/core/src/main/java/com/twitter/elephantbird/mapreduce/input/LzoInputFormat.java] is an example of the code that's broken because of this. - On the server side, FileDataServlet incorrectly sends SC_BAD_REQUEST instead of SC_NOT_FOUND for non-exitsing paths -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HDFS-6346) Optimize OP_SET_XATTRS by persisting single Xattr entry per setXattr/removeXattr api call
[ https://issues.apache.org/jira/browse/HDFS-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G resolved HDFS-6346. --- Resolution: Fixed Hadoop Flags: Reviewed I have just committed this to branch! Optimize OP_SET_XATTRS by persisting single Xattr entry per setXattr/removeXattr api call - Key: HDFS-6346 URL: https://issues.apache.org/jira/browse/HDFS-6346 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Uma Maheswara Rao G Assignee: Yi Liu Attachments: HDFS-6346.1.patch, HDFS-6346.2.patch, HDFS-6346.patch, editsStored When a new xattrs set on an Inode, it may add this with OP_SET_XATTRS and let's say [user.name1:value1] On a next call if we set another xattrs, then it may store along with older existing xattrs again. It may be like [user.name1:value1, user.name2:value2] So, on adding more xattrs on same Inode, that list may grow and we keep store the entries of already existing name, value fairs. Right now we defaulted the max Xattrs on an Inode to 32 and configured. If user modified it to much larger value and start setting xattrs, then edits loading may create issue like my above example. But I didn't refer any usecase of having large number of xattrs, this is just the assumption to consider a case. My biggest doubt is whether we will have such real usecases to have huge xattrs on a single INode. So, here is a thought on having OP_SET_XATTR for each setXAttr operation to be logged, When we replay them we need to consolidate. This is some initial thought we can think more if others also feel we need to consider this case to handle. Otherwise we endup storing Xattrs entries in editlog file as n(n+1)/2 where n is number xattrs for a file/dir. This may be issue only when we have large number configured max xattrs for inode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6373) Remove support for extended attributes on symlinks
[
https://issues.apache.org/jira/browse/HDFS-6373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995598#comment-13995598
]
Andrew Wang commented on HDFS-6373:
---
Related, I also found this page, which indicates that having ACLs on symlinks
is dubious:
http://0xced.blogspot.com/2009/03/chmod-acl-and-symbolic-links_23.html
This makes sense to me, since symlink permissions are always ignored. Linux
actually doesn't even let you change symlink permissions, though it is allowed
by the Unix spec. [~cnauroth], could you comment on whether this was
intentional, or just an oversight? It seems like we might want to remove
ACLs-on-symlinks, which is compatible since symlinks are still hard-disabled.
Remove support for extended attributes on symlinks
--
Key: HDFS-6373
URL: https://issues.apache.org/jira/browse/HDFS-6373
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Reporter: Andrew Wang
Looking in the Linux source code, we see the following:
http://lxr.linux.no/linux+v3.14.3/fs/xattr.c
{code}
60/*
61 * In the user.* namespace, only regular files and directories
can have
62 * extended attributes. For sticky directories, only the owner and
63 * privileged users can write attributes.
64 */
{code}
We should consider removing {{XAttrFeature}} from {{INodeSymlink}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HDFS-6398) Should have different limits for different XAttr namespaces (user, trusted, security and system).
Uma Maheswara Rao G created HDFS-6398: - Summary: Should have different limits for different XAttr namespaces (user, trusted, security and system). Key: HDFS-6398 URL: https://issues.apache.org/jira/browse/HDFS-6398 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Uma Maheswara Rao G Assignee: Yi Liu Nicholas pointed that, we should have different limits for different XAttr namespaces (user, trusted, security and system). Filing new JIRA by taking the comment from HDFS-2006 https://issues.apache.org/jira/browse/HDFS-2006?focusedCommentId=13997202page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13997202 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6399) FSNamesystem ACL operations should check isPermissionEnabled
[ https://issues.apache.org/jira/browse/HDFS-6399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6399: --- Attachment: HDFS-6399.1.patch FSNamesystem ACL operations should check isPermissionEnabled Key: HDFS-6399 URL: https://issues.apache.org/jira/browse/HDFS-6399 Project: Hadoop HDFS Issue Type: Bug Components: namenode Reporter: Charles Lamb Assignee: Charles Lamb Priority: Minor Attachments: HDFS-6399.1.patch The ACL operations in FSNamesystem don't currently check isPermissionEnabled before calling checkOwner(). This patch corrects that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6283) Write end user documentation for xattrs.
[ https://issues.apache.org/jira/browse/HDFS-6283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997558#comment-13997558 ] Charles Lamb commented on HDFS-6283: +1. Good catch Uma. Write end user documentation for xattrs. Key: HDFS-6283 URL: https://issues.apache.org/jira/browse/HDFS-6283 Project: Hadoop HDFS Issue Type: Sub-task Components: documentation Reporter: Chris Nauroth Assignee: Andrew Wang Fix For: HDFS XAttrs (HDFS-2006) Attachments: hdfs-6283-1.patch, hdfs-6283-2.patch, hdfs-6283-3.patch Update the File System Shell documentation to cover the new getfattr and setfattr commands. If warranted, consider adding a separate dedicated page for fuller discussion of the xattrs model and how the feature works in more detail. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6400) Cannot execute hdfs oiv_legacy
[
https://issues.apache.org/jira/browse/HDFS-6400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HDFS-6400:
Status: Patch Available (was: Open)
Cannot execute hdfs oiv_legacy
Key: HDFS-6400
URL: https://issues.apache.org/jira/browse/HDFS-6400
Project: Hadoop HDFS
Issue Type: Bug
Components: tools
Affects Versions: 2.5.0
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Priority: Critical
Labels: newbie
Attachments: HDFS-6400.patch
HDFS-6293 added hdfs oiv_legacy command to view a legacy fsimage, but
cannot execute the command.
In {{hdfs}},
{code}
elif [ COMMAND = oiv_legacy ] ; then
CLASS=org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewer
{code}
should be
{code}
elif [ $COMMAND = oiv_legacy ] ; then
CLASS=org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewer
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6305) WebHdfs response decoding may throw RuntimeExceptions
[ https://issues.apache.org/jira/browse/HDFS-6305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated HDFS-6305: -- Resolution: Fixed Fix Version/s: 2.5.0 3.0.0 Status: Resolved (was: Patch Available) With the +1 from [~cnauroth] and from [~hadoopqa], committing this to branch-2 and trunk. I'm also +1 on this issue. Thanks, [~daryn] WebHdfs response decoding may throw RuntimeExceptions - Key: HDFS-6305 URL: https://issues.apache.org/jira/browse/HDFS-6305 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Priority: Critical Fix For: 3.0.0, 2.5.0 Attachments: HDFS-6305.patch WebHdfs does not guard against exceptions while decoding the response payload. The json parser will throw RunTime exceptions on malformed responses. The json decoding routines do not validate the expected fields are present which may cause NPEs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6373) Remove support for extended attributes on symlinks
[
https://issues.apache.org/jira/browse/HDFS-6373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996992#comment-13996992
]
Andrew Wang commented on HDFS-6373:
---
+1 on the patch, thanks Charles.
Thanks for the confirmation also Chris, I guess this is really just for code
consistency for INodeSymlink. I filed HDFS-6396 and assigned it to Charles so
he can do a similar fix.
Remove support for extended attributes on symlinks
--
Key: HDFS-6373
URL: https://issues.apache.org/jira/browse/HDFS-6373
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Reporter: Andrew Wang
Assignee: Charles Lamb
Attachments: HDFS-6373.1.patch, HDFS-6373.2.patch
Looking in the Linux source code, we see the following:
http://lxr.linux.no/linux+v3.14.3/fs/xattr.c
{code}
60/*
61 * In the user.* namespace, only regular files and directories
can have
62 * extended attributes. For sticky directories, only the owner and
63 * privileged users can write attributes.
64 */
{code}
We should consider removing {{XAttrFeature}} from {{INodeSymlink}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6326) WebHdfs ACL compatibility is broken
[
https://issues.apache.org/jira/browse/HDFS-6326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997593#comment-13997593
]
Daryn Sharp commented on HDFS-6326:
---
I agree it's probably a good idea to keep FsAclPermission private. I'm still
+1, but arguably there's conflict of interest in me approving the code I helped
contribute. :) You may want to consider getting a second opinion but it sounds
like Haohui has offline approved the change?
WebHdfs ACL compatibility is broken
---
Key: HDFS-6326
URL: https://issues.apache.org/jira/browse/HDFS-6326
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 3.0.0, 2.4.0
Reporter: Daryn Sharp
Assignee: Chris Nauroth
Priority: Blocker
Attachments: HDFS-6326.1.patch, HDFS-6326.2.patch, HDFS-6326.3.patch,
HDFS-6326.4.patch, HDFS-6326.5.patch, aclfsperm.example
2.4 ACL support is completely incompatible with 2.4 webhdfs servers. The NN
throws an {{IllegalArgumentException}} exception.
{code}
hadoop fs -ls webhdfs://nn/
Found 21 items
ls: Invalid value for webhdfs parameter op: No enum constant
org.apache.hadoop.hdfs.web.resources.GetOpParam.Op.GETACLSTATUS
[... 20 more times...]
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6283) Write end user documentation for xattrs.
[ https://issues.apache.org/jira/browse/HDFS-6283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997154#comment-13997154 ] Charles Lamb commented on HDFS-6283: Nice. I have (as usual) a few minor nits below which are really optional. LGTM, +1. abbreviated as xattrs We should be careful that we are consistent throughout. I think I have seen it abbreviated as XAttrs in some places (like the code). Either way is fine with me, but we should be uniform. see the Linux manpage for attr(5) as well as the rest of the documentation). see the Linux attr manpage and related documentation). Xattrs must also specify a namespace = xattr names must also have a namespace prefix. These commands are styled after the Linux commands getfattr(1) and setfattr(1) - These commands are styled after the Linux getfattr(1) and setfattr(1) commands. Write end user documentation for xattrs. Key: HDFS-6283 URL: https://issues.apache.org/jira/browse/HDFS-6283 Project: Hadoop HDFS Issue Type: Sub-task Components: documentation Reporter: Chris Nauroth Assignee: Andrew Wang Attachments: hdfs-6283-1.patch, hdfs-6283-2.patch Update the File System Shell documentation to cover the new getfattr and setfattr commands. If warranted, consider adding a separate dedicated page for fuller discussion of the xattrs model and how the feature works in more detail. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6283) Write end user documentation for xattrs.
[ https://issues.apache.org/jira/browse/HDFS-6283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-6283: Hadoop Flags: Reviewed +1 for the patch. Thank you, Andrew. Write end user documentation for xattrs. Key: HDFS-6283 URL: https://issues.apache.org/jira/browse/HDFS-6283 Project: Hadoop HDFS Issue Type: Sub-task Components: documentation Reporter: Chris Nauroth Assignee: Andrew Wang Attachments: hdfs-6283-1.patch, hdfs-6283-2.patch Update the File System Shell documentation to cover the new getfattr and setfattr commands. If warranted, consider adding a separate dedicated page for fuller discussion of the xattrs model and how the feature works in more detail. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6340) DN can't finalize upgrade
[ https://issues.apache.org/jira/browse/HDFS-6340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13993473#comment-13993473 ] Rahul Singhal commented on HDFS-6340: - Thanks for commiting this [~arpitagarwal]. DN can't finalize upgrade - Key: HDFS-6340 URL: https://issues.apache.org/jira/browse/HDFS-6340 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 2.4.0 Reporter: Rahul Singhal Assignee: Rahul Singhal Priority: Blocker Fix For: 3.0.0, 2.4.1 Attachments: HDFS-6340-branch-2.4.0.patch, HDFS-6340.02.patch, HDFS-6340.patch I upgraded a (NN) HA cluster from 2.2.0 to 2.4.0. After I issued the '-finalizeUpgarde' command, NN was able to finalize the upgrade but DN couldn't (I waited for the next block report). I think I have found the problem to be due to HDFS-5153. I will attach a proposed fix. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-2949) HA: Add check to active state transition to prevent operator-induced split brain
[ https://issues.apache.org/jira/browse/HDFS-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated HDFS-2949: - Attachment: HDFS-2949-v3.patch HA: Add check to active state transition to prevent operator-induced split brain Key: HDFS-2949 URL: https://issues.apache.org/jira/browse/HDFS-2949 Project: Hadoop HDFS Issue Type: Improvement Components: ha, namenode Affects Versions: 0.24.0 Reporter: Todd Lipcon Assignee: Rushabh S Shah Attachments: HDFS-2949-v2.patch, HDFS-2949-v3.patch, HDFS-2949.patch Currently, if the administrator mistakenly calls -transitionToActive on one NN while the other one is still active, all hell will break loose. We can add a simple check by having the NN make a getServiceState() RPC to its peer with a short (~1 second?) timeout. If the RPC succeeds and indicates the other node is active, it should refuse to enter active mode. If the RPC fails or indicates standby, it can proceed. This is just meant as a preventative safety check - we still expect users to use the -failover command which has other checks plus fencing built in. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6283) Write end user documentation for xattrs.
[ https://issues.apache.org/jira/browse/HDFS-6283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6283: -- Attachment: hdfs-6283-3.patch Newest patch addresses Charlie's nits, will commit shortly. Thanks for reviewing all! Write end user documentation for xattrs. Key: HDFS-6283 URL: https://issues.apache.org/jira/browse/HDFS-6283 Project: Hadoop HDFS Issue Type: Sub-task Components: documentation Reporter: Chris Nauroth Assignee: Andrew Wang Attachments: hdfs-6283-1.patch, hdfs-6283-2.patch, hdfs-6283-3.patch Update the File System Shell documentation to cover the new getfattr and setfattr commands. If warranted, consider adding a separate dedicated page for fuller discussion of the xattrs model and how the feature works in more detail. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-2006) ability to support storing extended attributes per file
[ https://issues.apache.org/jira/browse/HDFS-2006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997753#comment-13997753 ] Andrew Wang commented on HDFS-2006: --- Related to limits, I also filed HDFS-6395 to enforce the # of xattrs per inode limit differently based on namespace. This could be blown out to per-NS settings, but I think it's sufficient to only enforce it for USER. I'll also note that many existing FS have arbitrary limits on the size of an xattr (e.g. page size) and that seems to work fine. I think it's unlikely we'd need to support large xattrs either, especially since (as Nicholas points out) we'd need to implement some special storage mechanism to avoid storing large xattrs with the rest of the FS metadata. If anything, I expect per-NS configs to be used to decrease the user limit. Related to this, we might also want to enforce a hardcoded *minimum* limit for the # and size of the non-user namespaces. This way, we could use xattrs for system features even if the admin has configured wacky settings. Being able to disable xattrs also makes it more annoying to rely on the system namespace, maybe we should remove that config option as well. With per-NS limits, the admin could still configure the user NS limits to 0 to prevent users from using xattrs. ability to support storing extended attributes per file --- Key: HDFS-2006 URL: https://issues.apache.org/jira/browse/HDFS-2006 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: HDFS XAttrs (HDFS-2006) Reporter: dhruba borthakur Assignee: Yi Liu Attachments: ExtendedAttributes.html, HDFS-XAttrs-Design-1.pdf, HDFS-XAttrs-Design-2.pdf, HDFS-XAttrs-Design-3.pdf, Test-Plan-for-Extended-Attributes-1.pdf, xattrs.1.patch, xattrs.patch It would be nice if HDFS provides a feature to store extended attributes for files, similar to the one described here: http://en.wikipedia.org/wiki/Extended_file_attributes. The challenge is that it has to be done in such a way that a site not using this feature does not waste precious memory resources in the namenode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6397) NN shows inconsistent value in deadnode count
Mohammad Kamrul Islam created HDFS-6397:
---
Summary: NN shows inconsistent value in deadnode count
Key: HDFS-6397
URL: https://issues.apache.org/jira/browse/HDFS-6397
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Mohammad Kamrul Islam
Assignee: Mohammad Kamrul Islam
Context:
When NN is started , without any live datanode but there are nodes in the
dfs.includes, NN shows the deadcount as '0'.
There are two inconsistencies:
1. If you click on deadnode links (which shows the count is 0), it will display
the list of deadnodes correctly.
2. hadoop 1.x used to display the count correctly.
The following snippets of JMX response will explain it further:
Look at the value of NumDeadDataNodes
{noformat}
{
name : Hadoop:service=NameNode,name=FSNamesystemState,
modelerType : org.apache.hadoop.hdfs.server.namenode.FSNamesystem,
CapacityTotal : 0,
CapacityUsed : 0,
...
NumLiveDataNodes : 0,
NumDeadDataNodes : 0,
NumDecomLiveDataNodes : 0,
NumDecomDeadDataNodes : 0,
NumDecommissioningDataNodes : 0,
NumStaleDataNodes : 0
},
{noformat}
Look at DeadNodes.
{noformat}
{
name : Hadoop:service=NameNode,name=NameNodeInfo,
modelerType : org.apache.hadoop.hdfs.server.namenode.FSNamesystem,
TotalBlocks : 70,
TotalFiles : 129,
NumberOfMissingBlocks : 0,
LiveNodes : {},
DeadNodes :
{\M.linkedin.com\:{\lastContact\:1400037397,\decommissioned\:false,\xferaddr\:\172.XX.X.XX:71\},\N.linkedin.com\:{\lastContact\:1400037397,\decommissioned\:false,\xferaddr\:\172.XX.XX.XX:71\}},
DecomNodes : {},
.
}
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6268) Better sorting in NetworkTopology#pseudoSortByDistance when no local node is found
[
https://issues.apache.org/jira/browse/HDFS-6268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996099#comment-13996099
]
Hadoop QA commented on HDFS-6268:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12644211/hdfs-6268-4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.TestNameNodeResourceChecker
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6886//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6886//console
This message is automatically generated.
Better sorting in NetworkTopology#pseudoSortByDistance when no local node is
found
--
Key: HDFS-6268
URL: https://issues.apache.org/jira/browse/HDFS-6268
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 2.4.0
Reporter: Andrew Wang
Assignee: Andrew Wang
Priority: Minor
Attachments: hdfs-6268-1.patch, hdfs-6268-2.patch, hdfs-6268-3.patch,
hdfs-6268-4.patch
In NetworkTopology#pseudoSortByDistance, if no local node is found, it will
always place the first rack local node in the list in front.
This became an issue when a dataset was loaded from a single datanode. This
datanode ended up being the first replica for all the blocks in the dataset.
When running an Impala query, the non-local reads when reading past a block
boundary were all hitting this node, meaning massive load skew.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6268) Better sorting in NetworkTopology#pseudoSortByDistance when no local node is found
[ https://issues.apache.org/jira/browse/HDFS-6268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6268: -- Attachment: hdfs-6268-4.patch Sorry for letting this set, new patch attached. I had some debugging printfs that were making various tests that looked at stdout fail, removed them. I also improved one of the failed tests while I was in there, it had the arguments in the asserts backwards. Better sorting in NetworkTopology#pseudoSortByDistance when no local node is found -- Key: HDFS-6268 URL: https://issues.apache.org/jira/browse/HDFS-6268 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 2.4.0 Reporter: Andrew Wang Assignee: Andrew Wang Priority: Minor Attachments: hdfs-6268-1.patch, hdfs-6268-2.patch, hdfs-6268-3.patch, hdfs-6268-4.patch In NetworkTopology#pseudoSortByDistance, if no local node is found, it will always place the first rack local node in the list in front. This became an issue when a dataset was loaded from a single datanode. This datanode ended up being the first replica for all the blocks in the dataset. When running an Impala query, the non-local reads when reading past a block boundary were all hitting this node, meaning massive load skew. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6326) WebHdfs ACL compatibility is broken
[
https://issues.apache.org/jira/browse/HDFS-6326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997813#comment-13997813
]
Kihwal Lee commented on HDFS-6326:
--
The precommit is still running. It ran common tests and
{{org.apache.hadoop.http.TestHttpServer}} timed out. You can speed up the
process by verifying it is not caused by this jira, while precommit is still
running.
Precommit is still running hdfs tests after common tests. I expect it to be
done around 18:35 UTC, which is less than an hour from now. I will check back
in an hour.
WebHdfs ACL compatibility is broken
---
Key: HDFS-6326
URL: https://issues.apache.org/jira/browse/HDFS-6326
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 3.0.0, 2.4.0
Reporter: Daryn Sharp
Assignee: Chris Nauroth
Priority: Blocker
Attachments: HDFS-6326.1.patch, HDFS-6326.2.patch, HDFS-6326.3.patch,
HDFS-6326.4.patch, HDFS-6326.5.patch, aclfsperm.example
2.4 ACL support is completely incompatible with 2.4 webhdfs servers. The NN
throws an {{IllegalArgumentException}} exception.
{code}
hadoop fs -ls webhdfs://nn/
Found 21 items
ls: Invalid value for webhdfs parameter op: No enum constant
org.apache.hadoop.hdfs.web.resources.GetOpParam.Op.GETACLSTATUS
[... 20 more times...]
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6326) WebHdfs ACL compatibility is broken
[
https://issues.apache.org/jira/browse/HDFS-6326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-6326:
Attachment: HDFS-6326.6.patch
Here is patch v6. I fixed the JavaDoc warning. I also added one more test in
{{FSAclBaseTest}} asserting that {{setPermission}} cannot mess with the ACL
bit. There is no problem related to this in the main code, because it uses
{{FsPermission#toShort}} to serialize the permissions, and we've made sure that
the ACL bit isn't visible through that method. I want this test in place
though to catch regressions in case that implementation detail ever changes.
For convenience, here is the incremental diff since last time, so reviewers
don't need to re-read the whole thing:
{code}
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/FsPermission.java
b/hadoop-common-
index c9fa89d..ee84437 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/FsPermission.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/FsPermission.java
@@ -159,7 +159,7 @@ public short toShort() {
}
/**
- * Encodes the object to a short. Unlike {@link toShort()}, this method may
+ * Encodes the object to a short. Unlike {@link #toShort()}, this method may
* return values outside the fixed range 0 - 01777 if extended features
* are encoded into this permission, such as the ACL bit.
*
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSAclBaseTest.java
b/hadoop-hdf
index 4aeeb85..f36483e 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSAclBaseTest.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSAclBaseTest.java
@@ -38,6 +38,7 @@
import org.apache.hadoop.hdfs.DFSTestUtil;
import org.apache.hadoop.hdfs.MiniDFSCluster;
import org.apache.hadoop.hdfs.protocol.AclException;
+import org.apache.hadoop.hdfs.protocol.FsAclPermission;
import org.apache.hadoop.io.IOUtils;
import org.apache.hadoop.security.AccessControlException;
import org.apache.hadoop.security.UserGroupInformation;
@@ -815,6 +816,23 @@ public void testSetPermissionOnlyDefault() throws
IOException {
}
@Test
+ public void testSetPermissionCannotSetAclBit() throws IOException {
+FileSystem.mkdirs(fs, path, FsPermission.createImmutable((short)0750));
+fs.setPermission(path, FsPermission.createImmutable((short)0700));
+assertPermission((short)0700);
+fs.setPermission(path,
+ new FsAclPermission(FsPermission.createImmutable((short)0755)));
+INode inode = cluster.getNamesystem().getFSDirectory().getNode(
+ path.toUri().getPath(), false);
+assertNotNull(inode);
+FsPermission perm = inode.getFsPermission();
+assertNotNull(perm);
+assertEquals(0755, perm.toShort());
+assertEquals(0755, perm.toExtendedShort());
+assertAclFeature(false);
+ }
+
+ @Test
public void testDefaultAclNewFile() throws Exception {
FileSystem.mkdirs(fs, path, FsPermission.createImmutable((short)0750));
ListAclEntry aclSpec = Lists.newArrayList(
{code}
WebHdfs ACL compatibility is broken
---
Key: HDFS-6326
URL: https://issues.apache.org/jira/browse/HDFS-6326
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 3.0.0, 2.4.0
Reporter: Daryn Sharp
Assignee: Chris Nauroth
Priority: Blocker
Attachments: HDFS-6326.1.patch, HDFS-6326.2.patch, HDFS-6326.3.patch,
HDFS-6326.4.patch, HDFS-6326.5.patch, HDFS-6326.6.patch, aclfsperm.example
2.4 ACL support is completely incompatible with 2.4 webhdfs servers. The NN
throws an {{IllegalArgumentException}} exception.
{code}
hadoop fs -ls webhdfs://nn/
Found 21 items
ls: Invalid value for webhdfs parameter op: No enum constant
org.apache.hadoop.hdfs.web.resources.GetOpParam.Op.GETACLSTATUS
[... 20 more times...]
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6326) WebHdfs ACL compatibility is broken
[
https://issues.apache.org/jira/browse/HDFS-6326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997624#comment-13997624
]
Kihwal Lee commented on HDFS-6326:
--
The patch looks good. I just kicked the precommit build.
WebHdfs ACL compatibility is broken
---
Key: HDFS-6326
URL: https://issues.apache.org/jira/browse/HDFS-6326
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 3.0.0, 2.4.0
Reporter: Daryn Sharp
Assignee: Chris Nauroth
Priority: Blocker
Attachments: HDFS-6326.1.patch, HDFS-6326.2.patch, HDFS-6326.3.patch,
HDFS-6326.4.patch, HDFS-6326.5.patch, aclfsperm.example
2.4 ACL support is completely incompatible with 2.4 webhdfs servers. The NN
throws an {{IllegalArgumentException}} exception.
{code}
hadoop fs -ls webhdfs://nn/
Found 21 items
ls: Invalid value for webhdfs parameter op: No enum constant
org.apache.hadoop.hdfs.web.resources.GetOpParam.Op.GETACLSTATUS
[... 20 more times...]
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6374) setXAttr should require the user to be the owner of the file or directory
[
https://issues.apache.org/jira/browse/HDFS-6374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996264#comment-13996264
]
Charles Lamb commented on HDFS-6374:
Good point Yi. We'll take that into account in the fix for this.
Charles
setXAttr should require the user to be the owner of the file or directory
-
Key: HDFS-6374
URL: https://issues.apache.org/jira/browse/HDFS-6374
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: HDFS XAttrs (HDFS-2006)
Reporter: Andrew Wang
Assignee: Charles Lamb
From the attr(5) manpage:
{noformat}
For this reason, extended user attributes are only allowed for regular
files and directories, and access to extended user attributes is
restricted to the owner and to users with appropriate capabilities for
directories with the sticky bit set (see the chmod(1) manual page for
an explanation of Sticky Directories).
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6326) WebHdfs ACL compatibility is broken
[
https://issues.apache.org/jira/browse/HDFS-6326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997918#comment-13997918
]
Kihwal Lee commented on HDFS-6326:
--
Probably we can have FsACLPermission#equals() simply call super.equals() to
make findbugs happy.
WebHdfs ACL compatibility is broken
---
Key: HDFS-6326
URL: https://issues.apache.org/jira/browse/HDFS-6326
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 3.0.0, 2.4.0
Reporter: Daryn Sharp
Assignee: Chris Nauroth
Priority: Blocker
Attachments: HDFS-6326.1.patch, HDFS-6326.2.patch, HDFS-6326.3.patch,
HDFS-6326.4.patch, HDFS-6326.5.patch, HDFS-6326.6.patch, aclfsperm.example
2.4 ACL support is completely incompatible with 2.4 webhdfs servers. The NN
throws an {{IllegalArgumentException}} exception.
{code}
hadoop fs -ls webhdfs://nn/
Found 21 items
ls: Invalid value for webhdfs parameter op: No enum constant
org.apache.hadoop.hdfs.web.resources.GetOpParam.Op.GETACLSTATUS
[... 20 more times...]
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6326) WebHdfs ACL compatibility is broken
[
https://issues.apache.org/jira/browse/HDFS-6326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997862#comment-13997862
]
Kihwal Lee commented on HDFS-6326:
--
Did you check the findbugs warning? Two different FsACLPermission objects can
have the same hash code and equal. Please address this issue and fix the
javadoc warning.
WebHdfs ACL compatibility is broken
---
Key: HDFS-6326
URL: https://issues.apache.org/jira/browse/HDFS-6326
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 3.0.0, 2.4.0
Reporter: Daryn Sharp
Assignee: Chris Nauroth
Priority: Blocker
Attachments: HDFS-6326.1.patch, HDFS-6326.2.patch, HDFS-6326.3.patch,
HDFS-6326.4.patch, HDFS-6326.5.patch, HDFS-6326.6.patch, aclfsperm.example
2.4 ACL support is completely incompatible with 2.4 webhdfs servers. The NN
throws an {{IllegalArgumentException}} exception.
{code}
hadoop fs -ls webhdfs://nn/
Found 21 items
ls: Invalid value for webhdfs parameter op: No enum constant
org.apache.hadoop.hdfs.web.resources.GetOpParam.Op.GETACLSTATUS
[... 20 more times...]
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6250) TestBalancerWithNodeGroup.testBalancerWithRackLocality fails
[
https://issues.apache.org/jira/browse/HDFS-6250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997583#comment-13997583
]
Chen He commented on HDFS-6250:
---
Hi [~djp], I think it is fine for now, [~decster]'s patch can resolve most of
the problems. There is corner case that this patch may fail.
Unit test is to verify the correctness of source code. This test case wants to
check whether balancer can keep the rack-locality if there is not over-utilized
nodes. This patch may report false negative in following condition:
If the balancer.id file is really large and make both node0 and node1 (two
nodes original in the minicluster before adding new nodes) over-utilized. The
balancer will move data blocks from rack0 to rack1. Because balancer's purpose
is to get rid of over-utilized and under-utilized nodes by moving blocks. This
patch will report failure. However, the balancer is not malfunctioned. If we
bring up two new nodes to the minicluster, one to rack0 and one to rack1, it
will be safe.
TestBalancerWithNodeGroup.testBalancerWithRackLocality fails
Key: HDFS-6250
URL: https://issues.apache.org/jira/browse/HDFS-6250
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Kihwal Lee
Assignee: Chen He
Attachments: HDFS-6250-v2.patch, HDFS-6250-v3.patch, HDFS-6250.patch,
test_log.txt
It was seen in https://builds.apache.org/job/PreCommit-HDFS-Build/6669/
{panel}
java.lang.AssertionError: expected:1800 but was:1810
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:147)
at org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
.testBalancerWithRackLocality(TestBalancerWithNodeGroup.java:253)
{panel}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6250) TestBalancerWithNodeGroup.testBalancerWithRackLocality fails
[
https://issues.apache.org/jira/browse/HDFS-6250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997684#comment-13997684
]
Junping Du commented on HDFS-6250:
--
Thanks for review and comments, [~airbots]!
The good case you proposed above, if it is current behavior, will be seen as a
bug for balancer. As the prerequisite of balancer's handling block movement is
not hurt the data reliability. It shouldn't move replicas of balancer.id across
rack0 and rack1 as it will make replica's rack number reduce to 1 which affect
block's reliability and inconsistent with replica placement policy. Actually,
we have code below to get rid of this case:
{code}
...
* 3. doing the move does not reduce the number of racks that the block has
*/
private boolean isGoodBlockCandidate(Source source,
{code}
Would you double check the behavior of balancer in the case you described
above? If so, we should file a separated JIRA to fix Balancer. What do you
think?
TestBalancerWithNodeGroup.testBalancerWithRackLocality fails
Key: HDFS-6250
URL: https://issues.apache.org/jira/browse/HDFS-6250
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Kihwal Lee
Assignee: Chen He
Attachments: HDFS-6250-v2.patch, HDFS-6250-v3.patch, HDFS-6250.patch,
test_log.txt
It was seen in https://builds.apache.org/job/PreCommit-HDFS-Build/6669/
{panel}
java.lang.AssertionError: expected:1800 but was:1810
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:147)
at org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
.testBalancerWithRackLocality(TestBalancerWithNodeGroup.java:253)
{panel}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-5522) Datanode disk error check may be incorrectly skipped
[
https://issues.apache.org/jira/browse/HDFS-5522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HDFS-5522:
-
Status: Patch Available (was: Open)
Submitting a new patch incorporating Kihwals comments.
Datanode disk error check may be incorrectly skipped
Key: HDFS-5522
URL: https://issues.apache.org/jira/browse/HDFS-5522
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.2.0, 0.23.9
Reporter: Kihwal Lee
Assignee: Rushabh S Shah
Attachments: HDFS-5522-v2.patch, HDFS-5522-v3.patch, HDFS-5522.patch
After HDFS-4581 and HDFS-4699, {{checkDiskError()}} is not called when
network errors occur during processing data node requests. This appears to
create problems when a disk is having problems, but not failing I/O soon.
If I/O hangs for a long time, network read/write may timeout first and the
peer may close the connection. Although the error was caused by a faulty
local disk, disk check is not being carried out in this case.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HDFS-6319) Various syntax and style cleanups
[
https://issues.apache.org/jira/browse/HDFS-6319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Charles Lamb resolved HDFS-6319.
Resolution: Won't Fix
Lack of interest.
Various syntax and style cleanups
-
Key: HDFS-6319
URL: https://issues.apache.org/jira/browse/HDFS-6319
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Charles Lamb
Assignee: Charles Lamb
Attachments: HDFS-6319.1.patch, HDFS-6319.2.patch, HDFS-6319.3.patch,
HDFS-6319.4.patch, HDFS-6319.6.patch, HDFS-6319.7.patch, HDFS-6319.8.patch,
HDFS-6319.8.patch
Fix various style issues like if(, while(, [i.e. lack of a space after the
keyword],
Extra whitespace and newlines
if (...) return ... [lack of {}'s]
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6326) WebHdfs ACL compatibility is broken
[
https://issues.apache.org/jira/browse/HDFS-6326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995494#comment-13995494
]
Daryn Sharp commented on HDFS-6326:
---
bq. failing to justify the performance concerns leads Chris and I to decide to
keep ACL out of the critical path, and to move the complexity to ls / web UI.
Performance concern: The additional rpc and/or http calls. I misspoke when I
had 2X the calls. It's 1 listStatus per directory, N-many getAclStatus per
item just to decide whether to print a + or a . Let's take a real world
example from a busy cluster: 200ms queue time, 2ms response time. Listing a
directory with 100 items will take an extra ~20s. Listing 1000 items will take
an extra ~3.4m. We have users that ls larger datasets.
This is unacceptable.
WebHdfs ACL compatibility is broken
---
Key: HDFS-6326
URL: https://issues.apache.org/jira/browse/HDFS-6326
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 3.0.0, 2.4.0
Reporter: Daryn Sharp
Assignee: Chris Nauroth
Priority: Blocker
Attachments: HDFS-6326.1.patch, HDFS-6326.2.patch, HDFS-6326.3.patch
2.4 ACL support is completely incompatible with 2.4 webhdfs servers. The NN
throws an {{IllegalArgumentException}} exception.
{code}
hadoop fs -ls webhdfs://nn/
Found 21 items
ls: Invalid value for webhdfs parameter op: No enum constant
org.apache.hadoop.hdfs.web.resources.GetOpParam.Op.GETACLSTATUS
[... 20 more times...]
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6377) Unify xattr name and value limits into a single limit
[
https://issues.apache.org/jira/browse/HDFS-6377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996982#comment-13996982
]
Chris Nauroth commented on HDFS-6377:
-
The patch looked good, Andrew. Thanks! Maybe a few potential cleanups:
{code}
+import java.io.UnsupportedEncodingException;
{code}
Is the import unused now?
{code}
- cacheManager.stopMonitorThread();
- cacheManager.clearDirectiveStats();
+ if (cacheManager != null) {
+cacheManager.stopMonitorThread();
+cacheManager.clearDirectiveStats();
+ }
{code}
Was this an unrelated change?
Unify xattr name and value limits into a single limit
-
Key: HDFS-6377
URL: https://issues.apache.org/jira/browse/HDFS-6377
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: HDFS XAttrs (HDFS-2006)
Reporter: Andrew Wang
Assignee: Andrew Wang
Fix For: HDFS XAttrs (HDFS-2006)
Attachments: hdfs-6377-1.patch, hdfs-6377-2.patch
Instead of having separate limits and config options for the size of an
xattr's name and value, let's use a single limit.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6326) WebHdfs ACL compatibility is broken
[
https://issues.apache.org/jira/browse/HDFS-6326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13993676#comment-13993676
]
Chris Nauroth commented on HDFS-6326:
-
Earlier comments are correct that there is no concern about distcp. It only
preserves ACLs if you specify a new option, so you don't pay a cost if you're
not using ACLs. (This is actually irrelevant to the discussion of 2.4.1
anyway. The distcp -pa patch is uncommitted and currently targeting 2.5.0.)
bq. It wouldn't persist the bit in the mask, nor serialize it to ensure
complete client compatibility.
[~daryn], if I understand your proposal completely:
# {{FsPermission#fromShort}} would recognize the ACL bit and set the private
{{aclBit}} member to true. This way, the client can see an ACL bit.
# {{FsPermission#toShort}} would ignore the private {{aclBit}} member. This is
to keep values in the range -01777.
# {{FsPermission#write}} would ignore the private {{aclBit}} member. This
follows from #2, since {{write}} is implemented in terms of {{toShort}}, and
once again it's done to keep visible values in the range -01777.
# {{FsAclPermission}} would be a class in HDFS (not Common), and it would be
the only thing capable of serializing the ACL bit.
Am I understanding your proposal correctly? My concern with this is that it
creates a confusing API on the client side, because there isn't symmetry
between serialization and deserialization. Round-tripping an {{FsPermission}}
instance would lead to data loss.
BTW, WebHDFS responses also include the octal representation of
{{FsPermission}}. If we don't want the ACL bit visible there, then we'd need
to address that separately. This probably means a separate aclBit field in
the JSON response and a client-side special path to look for aclBit in the
JSON and then toggle it on before calling {{FsPermission#fromShort}}.
But is it really necessary to restrict the client-visible range to -1777?
Is the problem theoretical or real? Do we know any real clients that are
directly referencing the {{toShort}} value? The only valid way to interpret
that data would be bitmask operations, and if any client is looking at the
proposed ACL bit today, then they are seeing undefined behavior. (IOW, the
client code is buggy.) I'd prefer not to incur a lot of code complexity
restricting the range to -1777, but if there is no other choice, then I'm
willing to compromise. Let's just consider if it's really necessary.
WebHdfs ACL compatibility is broken
---
Key: HDFS-6326
URL: https://issues.apache.org/jira/browse/HDFS-6326
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 3.0.0, 2.4.0
Reporter: Daryn Sharp
Assignee: Chris Nauroth
Priority: Blocker
Attachments: HDFS-6326.1.patch, HDFS-6326.2.patch, HDFS-6326.3.patch
2.4 ACL support is completely incompatible with 2.4 webhdfs servers. The NN
throws an {{IllegalArgumentException}} exception.
{code}
hadoop fs -ls webhdfs://nn/
Found 21 items
ls: Invalid value for webhdfs parameter op: No enum constant
org.apache.hadoop.hdfs.web.resources.GetOpParam.Op.GETACLSTATUS
[... 20 more times...]
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-2949) HA: Add check to active state transition to prevent operator-induced split brain
[ https://issues.apache.org/jira/browse/HDFS-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13998010#comment-13998010 ] Rushabh S Shah commented on HDFS-2949: -- Thanks Kihwal for reviewing and committing the patch. HA: Add check to active state transition to prevent operator-induced split brain Key: HDFS-2949 URL: https://issues.apache.org/jira/browse/HDFS-2949 Project: Hadoop HDFS Issue Type: Improvement Components: ha, namenode Affects Versions: 0.24.0 Reporter: Todd Lipcon Assignee: Rushabh S Shah Fix For: 3.0.0, 2.5.0 Attachments: HDFS-2949-v2.patch, HDFS-2949-v3.patch, HDFS-2949.patch Currently, if the administrator mistakenly calls -transitionToActive on one NN while the other one is still active, all hell will break loose. We can add a simple check by having the NN make a getServiceState() RPC to its peer with a short (~1 second?) timeout. If the RPC succeeds and indicates the other node is active, it should refuse to enter active mode. If the RPC fails or indicates standby, it can proceed. This is just meant as a preventative safety check - we still expect users to use the -failover command which has other checks plus fencing built in. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6283) Write end user documentation for xattrs.
[
https://issues.apache.org/jira/browse/HDFS-6283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996743#comment-13996743
]
Uma Maheswara Rao G commented on HDFS-6283:
---
Thanks a lot Andrew for the documentation!
Here are my quick comments on the patch!
dfs.namenode.inode.xattrs.max-limits -- dfs.namenode.inode.xattrs.max-limit
{code}
* dfs.namenode.inode.xattrs.name.max-length
+
+ The maximum length of the name of an extended attribute. By default, this
limit is 64 characters.
+
+ * dfs.namenode.inode.xattrs.value.max-length
+
+ The maximum length of the value of an extended attribute. By default, this
limit is 16384 bytes.
{code}
Please update as per JIRA HDFS-6377
Also need update in hdfs-default.xml
XAttrCommands is from common code, so can you leave that changes in that file
here. May we can handle that improvement as part of HADOOP-10561?
Dump all extended *attribute* values associated with pathname -- Dump all
values of the extended *attributes* associated with pathname
Displays the extended *attribute* names and values -- Displays the names and
values of the extended *attributes* ?
Write end user documentation for xattrs.
Key: HDFS-6283
URL: https://issues.apache.org/jira/browse/HDFS-6283
Project: Hadoop HDFS
Issue Type: Sub-task
Components: documentation
Reporter: Chris Nauroth
Assignee: Andrew Wang
Attachments: hdfs-6283-1.patch
Update the File System Shell documentation to cover the new getfattr and
setfattr commands. If warranted, consider adding a separate dedicated page
for fuller discussion of the xattrs model and how the feature works in more
detail.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HDFS-6344) Maximum limit on the size of an xattr
[ https://issues.apache.org/jira/browse/HDFS-6344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G resolved HDFS-6344. --- Resolution: Fixed Fix Version/s: HDFS XAttrs (HDFS-2006) Hadoop Flags: Reviewed Committed to branch! Maximum limit on the size of an xattr - Key: HDFS-6344 URL: https://issues.apache.org/jira/browse/HDFS-6344 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: HDFS XAttrs (HDFS-2006) Reporter: Andrew Wang Assignee: Yi Liu Fix For: HDFS XAttrs (HDFS-2006) Attachments: HDFS-6344.patch We should support limits on the maximum size of an xattr name/value. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (HDFS-6398) Should have different size limits for different XAttr namespaces (user, trusted, security and system).
[ https://issues.apache.org/jira/browse/HDFS-6398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997885#comment-13997885 ] Uma Maheswara Rao G edited comment on HDFS-6398 at 5/14/14 6:52 PM: I did not notice Andrew has filed a JIRA to cover this already. Closing this as duplicate. was (Author: umamaheswararao): I did not notice Andrew has file a JIRA to cover this already. Closing this as duplicate. Should have different size limits for different XAttr namespaces (user, trusted, security and system). -- Key: HDFS-6398 URL: https://issues.apache.org/jira/browse/HDFS-6398 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Uma Maheswara Rao G Assignee: Yi Liu Nicholas pointed that, we should have different limits for different XAttr namespaces (user, trusted, security and system). Filing new JIRA by taking the comment from HDFS-2006 https://issues.apache.org/jira/browse/HDFS-2006?focusedCommentId=13997202page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13997202 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-2949) HA: Add check to active state transition to prevent operator-induced split brain
[
https://issues.apache.org/jira/browse/HDFS-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997928#comment-13997928
]
Hadoop QA commented on HDFS-2949:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12644835/HDFS-2949-v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6899//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6899//console
This message is automatically generated.
HA: Add check to active state transition to prevent operator-induced split
brain
Key: HDFS-2949
URL: https://issues.apache.org/jira/browse/HDFS-2949
Project: Hadoop HDFS
Issue Type: Improvement
Components: ha, namenode
Affects Versions: 0.24.0
Reporter: Todd Lipcon
Assignee: Rushabh S Shah
Attachments: HDFS-2949-v2.patch, HDFS-2949-v3.patch, HDFS-2949.patch
Currently, if the administrator mistakenly calls -transitionToActive on one
NN while the other one is still active, all hell will break loose. We can add
a simple check by having the NN make a getServiceState() RPC to its peer with
a short (~1 second?) timeout. If the RPC succeeds and indicates the other
node is active, it should refuse to enter active mode. If the RPC fails or
indicates standby, it can proceed.
This is just meant as a preventative safety check - we still expect users to
use the -failover command which has other checks plus fencing built in.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-2949) HA: Add check to active state transition to prevent operator-induced split brain
[ https://issues.apache.org/jira/browse/HDFS-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13998002#comment-13998002 ] Kihwal Lee commented on HDFS-2949: -- +1 The patch looks good. HA: Add check to active state transition to prevent operator-induced split brain Key: HDFS-2949 URL: https://issues.apache.org/jira/browse/HDFS-2949 Project: Hadoop HDFS Issue Type: Improvement Components: ha, namenode Affects Versions: 0.24.0 Reporter: Todd Lipcon Assignee: Rushabh S Shah Attachments: HDFS-2949-v2.patch, HDFS-2949-v3.patch, HDFS-2949.patch Currently, if the administrator mistakenly calls -transitionToActive on one NN while the other one is still active, all hell will break loose. We can add a simple check by having the NN make a getServiceState() RPC to its peer with a short (~1 second?) timeout. If the RPC succeeds and indicates the other node is active, it should refuse to enter active mode. If the RPC fails or indicates standby, it can proceed. This is just meant as a preventative safety check - we still expect users to use the -failover command which has other checks plus fencing built in. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6250) TestBalancerWithNodeGroup.testBalancerWithRackLocality fails
[
https://issues.apache.org/jira/browse/HDFS-6250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997412#comment-13997412
]
Junping Du commented on HDFS-6250:
--
Thanks [~decster] for updating on this.
[~airbots], I will commit it if you are also fine with the patch here.
TestBalancerWithNodeGroup.testBalancerWithRackLocality fails
Key: HDFS-6250
URL: https://issues.apache.org/jira/browse/HDFS-6250
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Kihwal Lee
Assignee: Chen He
Attachments: HDFS-6250-v2.patch, HDFS-6250-v3.patch, HDFS-6250.patch,
test_log.txt
It was seen in https://builds.apache.org/job/PreCommit-HDFS-Build/6669/
{panel}
java.lang.AssertionError: expected:1800 but was:1810
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:147)
at org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
.testBalancerWithRackLocality(TestBalancerWithNodeGroup.java:253)
{panel}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6329) WebHdfs does not work if HA is enabled on NN but logical URI is not configured.
[ https://issues.apache.org/jira/browse/HDFS-6329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13992932#comment-13992932 ] Kihwal Lee commented on HDFS-6329: -- bq. We should really sort things out and make test cases do the right thing instead of relying on side-effects of shared conf \[ in MiniDFSCluster\]. But that's for another day and I am simply making it behave like the old way for now. I saw unit test failures in other JIRAs due to this issue, so I filed HDFS-6360. WebHdfs does not work if HA is enabled on NN but logical URI is not configured. --- Key: HDFS-6329 URL: https://issues.apache.org/jira/browse/HDFS-6329 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.4.0 Reporter: Kihwal Lee Assignee: Kihwal Lee Priority: Blocker Attachments: HDFS-6329.patch, HDFS-6329.patch, HDFS-6329.v2.patch, HDFS-6329.v3.patch, HDFS-6329.v4.patch After HDFS-6100, namenode unconditionally puts the logical name (name service id) as the token service when redirecting webhdfs requests to datanodes, if it detects HA. For HA configurations with no client-side failover proxy provider (e.g. IP failover), webhdfs does not work since the clients do not use logical name. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6357) SetXattr should persist rpcIDs for handling Namenode restart and HA
[ https://issues.apache.org/jira/browse/HDFS-6357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-6357: -- Description: SetXattr should persist rpcIDs for handling restart Namenode and HA scenarios (was: SetXattrs should persist rpcIDs for handling restart Namenode and HA scenarios) SetXattr should persist rpcIDs for handling Namenode restart and HA --- Key: HDFS-6357 URL: https://issues.apache.org/jira/browse/HDFS-6357 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: HDFS XAttrs (HDFS-2006) Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G Attachments: HDFS-6357.patch SetXattr should persist rpcIDs for handling restart Namenode and HA scenarios -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6328) Simplify code in FSDirectory
[ https://issues.apache.org/jira/browse/HDFS-6328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13993823#comment-13993823 ] Haohui Mai commented on HDFS-6328: -- Attaching v4 patch to address Jing's comments. [~daryn], do you have any more comments? I'll commit it later today if there is no more comment. Simplify code in FSDirectory Key: HDFS-6328 URL: https://issues.apache.org/jira/browse/HDFS-6328 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Haohui Mai Assignee: Haohui Mai Attachments: HDFS-6328.000.patch, HDFS-6328.001.patch, HDFS-6328.002.patch, HDFS-6328.003.patch, HDFS-6328.004.patch This jira proposes: # Cleaning up dead code in FSDirectory. # Simplify the control flows that IntelliJ flags as warnings. # Move functions related to resolving paths into one place. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6374) setXAttr should require the user to be the owner of the file or directory
[
https://issues.apache.org/jira/browse/HDFS-6374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Charles Lamb updated HDFS-6374:
---
Attachment: HDFS-6374.2.patch
Corrected to not call checkOwner() during getXAttr.
setXAttr should require the user to be the owner of the file or directory
-
Key: HDFS-6374
URL: https://issues.apache.org/jira/browse/HDFS-6374
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: HDFS XAttrs (HDFS-2006)
Reporter: Andrew Wang
Assignee: Charles Lamb
Attachments: HDFS-6374.1.patch, HDFS-6374.2.patch
From the attr(5) manpage:
{noformat}
For this reason, extended user attributes are only allowed for regular
files and directories, and access to extended user attributes is
restricted to the owner and to users with appropriate capabilities for
directories with the sticky bit set (see the chmod(1) manual page for
an explanation of Sticky Directories).
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6340) DN can't finalize upgarde
[ https://issues.apache.org/jira/browse/HDFS-6340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13993122#comment-13993122 ] Arpit Agarwal commented on HDFS-6340: - +1 pending Jenkins results. DN can't finalize upgarde - Key: HDFS-6340 URL: https://issues.apache.org/jira/browse/HDFS-6340 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 2.4.0 Reporter: Rahul Singhal Priority: Blocker Fix For: 3.0.0, 2.4.1 Attachments: HDFS-6340-branch-2.4.0.patch, HDFS-6340.02.patch, HDFS-6340.patch I upgraded a (NN) HA cluster from 2.2.0 to 2.4.0. After I issued the '-finalizeUpgarde' command, NN was able to finalize the upgrade but DN couldn't (I waited for the next block report). I think I have found the problem to be due to HDFS-5153. I will attach a proposed fix. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HDFS-6375) Listing extended attributes with the search permission
[
https://issues.apache.org/jira/browse/HDFS-6375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Charles Lamb reassigned HDFS-6375:
--
Assignee: Charles Lamb
Listing extended attributes with the search permission
--
Key: HDFS-6375
URL: https://issues.apache.org/jira/browse/HDFS-6375
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: HDFS XAttrs (HDFS-2006)
Reporter: Andrew Wang
Assignee: Charles Lamb
From the attr(5) manpage:
{noformat}
Users with search access to a file or directory may retrieve a list of
attribute names defined for that file or directory.
{noformat}
This is like doing {{getfattr}} without the {{-d}} flag, which we currently
don't support.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6305) WebHdfs response decoding may throw RuntimeExceptions
[ https://issues.apache.org/jira/browse/HDFS-6305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated HDFS-6305: -- Status: Patch Available (was: Open) Trying again to kick the precommit. WebHdfs response decoding may throw RuntimeExceptions - Key: HDFS-6305 URL: https://issues.apache.org/jira/browse/HDFS-6305 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Priority: Critical Attachments: HDFS-6305.patch WebHdfs does not guard against exceptions while decoding the response payload. The json parser will throw RunTime exceptions on malformed responses. The json decoding routines do not validate the expected fields are present which may cause NPEs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6398) Should have different size limits for different XAttr namespaces (user, trusted, security and system).
[ https://issues.apache.org/jira/browse/HDFS-6398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-6398: -- Summary: Should have different size limits for different XAttr namespaces (user, trusted, security and system). (was: Should have different limits for different XAttr namespaces (user, trusted, security and system).) Should have different size limits for different XAttr namespaces (user, trusted, security and system). -- Key: HDFS-6398 URL: https://issues.apache.org/jira/browse/HDFS-6398 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Uma Maheswara Rao G Assignee: Yi Liu Nicholas pointed that, we should have different limits for different XAttr namespaces (user, trusted, security and system). Filing new JIRA by taking the comment from HDFS-2006 https://issues.apache.org/jira/browse/HDFS-2006?focusedCommentId=13997202page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13997202 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-2949) HA: Add check to active state transition to prevent operator-induced split brain
[ https://issues.apache.org/jira/browse/HDFS-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997723#comment-13997723 ] Kihwal Lee commented on HDFS-2949: -- I've manually kicked precommit. HA: Add check to active state transition to prevent operator-induced split brain Key: HDFS-2949 URL: https://issues.apache.org/jira/browse/HDFS-2949 Project: Hadoop HDFS Issue Type: Improvement Components: ha, namenode Affects Versions: 0.24.0 Reporter: Todd Lipcon Assignee: Rushabh S Shah Attachments: HDFS-2949-v2.patch, HDFS-2949-v3.patch, HDFS-2949.patch Currently, if the administrator mistakenly calls -transitionToActive on one NN while the other one is still active, all hell will break loose. We can add a simple check by having the NN make a getServiceState() RPC to its peer with a short (~1 second?) timeout. If the RPC succeeds and indicates the other node is active, it should refuse to enter active mode. If the RPC fails or indicates standby, it can proceed. This is just meant as a preventative safety check - we still expect users to use the -failover command which has other checks plus fencing built in. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6361) TestIdUserGroup.testUserUpdateSetting failed due to out of range nfsnobody Id
[ https://issues.apache.org/jira/browse/HDFS-6361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996944#comment-13996944 ] Brandon Li commented on HDFS-6361: -- +1. Pending jenkins. TestIdUserGroup.testUserUpdateSetting failed due to out of range nfsnobody Id - Key: HDFS-6361 URL: https://issues.apache.org/jira/browse/HDFS-6361 Project: Hadoop HDFS Issue Type: Bug Components: nfs Affects Versions: 2.4.0 Reporter: Yongjun Zhang Assignee: Yongjun Zhang Attachments: HDFS-6361.001.patch, HDFS-6361.002.patch, HDFS-6361.003.patch The following error happens pretty often: org.apache.hadoop.nfs.nfs3.TestIdUserGroup.testUserUpdateSetting Failing for the past 1 build (Since Unstable#61 ) Took 0.1 sec. add description Error Message For input string: 4294967294 Stacktrace java.lang.NumberFormatException: For input string: 4294967294 at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.lang.Integer.parseInt(Integer.java:495) at java.lang.Integer.valueOf(Integer.java:582) at org.apache.hadoop.nfs.nfs3.IdUserGroup.updateMapInternal(IdUserGroup.java:137) at org.apache.hadoop.nfs.nfs3.IdUserGroup.updateMaps(IdUserGroup.java:188) at org.apache.hadoop.nfs.nfs3.IdUserGroup.init(IdUserGroup.java:60) at org.apache.hadoop.nfs.nfs3.TestIdUserGroup.testUserUpdateSetting(TestIdUserGroup.java:71) Standard Output log4j:WARN No appenders could be found for logger (org.apache.hadoop.nfs.nfs3.IdUserGroup). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6376) Distcp data between two HA clusters requires another configuration
[ https://issues.apache.org/jira/browse/HDFS-6376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Marion updated HDFS-6376: -- Target Version/s: 2.4.1 Status: Patch Available (was: Open) Distcp data between two HA clusters requires another configuration -- Key: HDFS-6376 URL: https://issues.apache.org/jira/browse/HDFS-6376 Project: Hadoop HDFS Issue Type: Bug Components: datanode, federation, hdfs-client Affects Versions: 2.4.0, 2.3.0 Environment: CDH 5.0 (Apache 2.3.0 + some patches) Reporter: Dave Marion Fix For: 2.4.1 Attachments: HDFS-6376-2.patch, HDFS-6376-patch-1.patch User has to create a third set of configuration files for distcp when transferring data between two HA clusters. Consider the scenario in [1]. You cannot put all of the required properties in core-site.xml and hdfs-site.xml for the client to resolve the location of both active namenodes. If you do, then the datanodes from cluster A may join cluster B. I can not find a configuration option that tells the datanodes to federate blocks for only one of the clusters in the configuration. [1] http://mail-archives.apache.org/mod_mbox/hadoop-user/201404.mbox/%3CBAY172-W2133964E0C283968C161DD1520%40phx.gbl%3E -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6056) Clean up NFS config settings
[ https://issues.apache.org/jira/browse/HDFS-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-6056: - Description: As discussed on HDFS-6050, there's a few opportunities to improve the config settings related to NFS. This JIRA is to implement those changes, which include: moving hdfs-nfs related properties into hadoop-hdfs-nfs project, and replacing 'nfs3' with 'nfs' in the property names. (was: As discussed on HDFS-6050, there's a few opportunities to improve the config settings related to NFS. This JIRA is to implement those changes, which include: moving hdfs-nfs related properties into hadoop-hdfs-nfs project, and replacing 'nfs' with 'nfs' in the property names.) Clean up NFS config settings Key: HDFS-6056 URL: https://issues.apache.org/jira/browse/HDFS-6056 Project: Hadoop HDFS Issue Type: Bug Components: nfs Affects Versions: 2.3.0 Reporter: Aaron T. Myers Assignee: Brandon Li Attachments: HDFS-6056.001.patch, HDFS-6056.002.patch, HDFS-6056.003.patch, HDFS-6056.004.patch As discussed on HDFS-6050, there's a few opportunities to improve the config settings related to NFS. This JIRA is to implement those changes, which include: moving hdfs-nfs related properties into hadoop-hdfs-nfs project, and replacing 'nfs3' with 'nfs' in the property names. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6367) EnumSetParam$Domain#parse fails for parameter containing more than one enum.
[
https://issues.apache.org/jira/browse/HDFS-6367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13998234#comment-13998234
]
Hudson commented on HDFS-6367:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #1753 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1753/])
HDFS-6367. EnumSetParam$Domain#parse fails for parameter containing more than
one enum. Contributed by Yi Liu. (umamahesh:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1594150)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/EnumSetParam.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/resources/TestParam.java
EnumSetParam$Domain#parse fails for parameter containing more than one enum.
Key: HDFS-6367
URL: https://issues.apache.org/jira/browse/HDFS-6367
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 3.0.0
Reporter: Yi Liu
Assignee: Yi Liu
Fix For: 3.0.0, 2.5.0
Attachments: HDFS-6367.patch
Fails because additional ,
{noformat}
java.lang.IllegalArgumentException: No enum const class
org.apache.hadoop.fs.Options$Rename.,OVERWRITE
at java.lang.Enum.valueOf(Enum.java:196)
at
org.apache.hadoop.hdfs.web.resources.EnumSetParam$Domain.parse(EnumSetParam.java:85)
at
org.apache.hadoop.hdfs.web.resources.RenameOptionSetParam.init(RenameOptionSetParam.java:45)
at
org.apache.hadoop.hdfs.web.resources.TestParam.testRenameOptionSetParam(TestParam.java:355)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6404) HttpFS should use a 000 umask for mkdir and create operations
[
https://issues.apache.org/jira/browse/HDFS-6404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13998253#comment-13998253
]
Alejandro Abdelnur commented on HDFS-6404:
--
The fix should be done in the {{FileSystemAccessService.java}} class in the
{{getFileSystemConfiguration()}} method, the configuration should be seeded
with {{conf.set(FsPermission.UMASK_LABEL, 000)}}
HttpFS should use a 000 umask for mkdir and create operations
-
Key: HDFS-6404
URL: https://issues.apache.org/jira/browse/HDFS-6404
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
The FileSystem created by HttpFS should use a 000 umask not to affect the
permissions set by the client as it is the responsibility of the client to
resolve the right permissions based on the client unmask.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6355) Fix divide-by-zero, improper use of wall-clock time in BlockPoolSliceScanner
[
https://issues.apache.org/jira/browse/HDFS-6355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13992493#comment-13992493
]
Hadoop QA commented on HDFS-6355:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12643860/HDFS-6355.001.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6856//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6856//console
This message is automatically generated.
Fix divide-by-zero, improper use of wall-clock time in BlockPoolSliceScanner
Key: HDFS-6355
URL: https://issues.apache.org/jira/browse/HDFS-6355
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HDFS-6355.001.patch
BlockPoolSliceScanner uses {{Time.now}} to calculate an interval. But this
is incorrect, since if the wall-clock time changes, we will end up setting
the scan periods to a shorter or longer time than we configured.
There is also a case where we may divide by zero if we get unlucky, because
we calculate an interval and divide by it, without checking whether the
interval is 0 milliseconds. This would produce an {{ArithmeticException}}
since we are using longs.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
