[jira] [Created] (HDFS-6391) Get the Key/IV from the NameNode for encrypted files in DFSClient
Alejandro Abdelnur created HDFS-6391: Summary: Get the Key/IV from the NameNode for encrypted files in DFSClient Key: HDFS-6391 URL: https://issues.apache.org/jira/browse/HDFS-6391 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode, security Reporter: Alejandro Abdelnur Assignee: Charles Lamb When creating/opening and encrypted file, the DFSClient should get the encryption key material and the IV for the file in the create/open RPC call. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HDFS-3278) Umbrella Jira for HDFS-HA Phase 2
[ https://issues.apache.org/jira/browse/HDFS-3278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved HDFS-3278. --- Resolution: Fixed Fix Version/s: 2.1.0-beta Assignee: Todd Lipcon (was: Sanjay Radia) These subtasks were completed quite a while back. Umbrella Jira for HDFS-HA Phase 2 - Key: HDFS-3278 URL: https://issues.apache.org/jira/browse/HDFS-3278 Project: Hadoop HDFS Issue Type: New Feature Reporter: Sanjay Radia Assignee: Todd Lipcon Fix For: 2.1.0-beta HDFS-1623 gives a high level architecture and design for hot automatic failover of the NN. Branch HDFS-1623 was merged into trunk for tactical reasons even though the work for HA was not complete, Branch HDFS-1623 contained mechanisms for keeping a standby Hot (ie read from shared journal), dual block reports, fencing of DNs, Zookeeper library for leader election etc. This Umbrella jira covers the remaining work for HA and will link all the jiras for the remaining work. Unlike HDFS-1623 no single branch will be created - work will proceed in parallel branches. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6408) Redundant definitions in log4j.properties
Abhiraj Butala created HDFS-6408:
Summary: Redundant definitions in log4j.properties
Key: HDFS-6408
URL: https://issues.apache.org/jira/browse/HDFS-6408
Project: Hadoop HDFS
Issue Type: Test
Reporter: Abhiraj Butala
Priority: Minor
Following definitions in
'hadoop-hdfs/src/contrib/bkjournal/src/test/resources/log4j.properties' are
defined twice and should be removed:
{code}
log4j.appender.ROLLINGFILE.layout=org.apache.log4j.PatternLayout
log4j.appender.ROLLINGFILE.layout.ConversionPattern=%d{ISO8601} - %-5p
[%t:%C{1}@%L] - %m%n
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HDFS-6417) TestTransferFsImage.testClientSideException is flaky
Binglin Chang created HDFS-6417: --- Summary: TestTransferFsImage.testClientSideException is flaky Key: HDFS-6417 URL: https://issues.apache.org/jira/browse/HDFS-6417 Project: Hadoop HDFS Issue Type: Bug Reporter: Binglin Chang Priority: Minor Attachments: HDFS-6417.log Looks like http connection to NN timeout. https://builds.apache.org/job/Hadoop-Hdfs-trunk/1753/testReport/org.apache.hadoop.hdfs.server.namenode/TestTransferFsImage/testClientSideException/ Error Message Wanted but not invoked: nNStorage.reportErrorOnFile( /x-does-not-exist/blah ); - at org.apache.hadoop.hdfs.server.namenode.TestTransferFsImage.testClientSideException(TestTransferFsImage.java:80) Actually, there were zero interactions with this mock. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6407) new namenode UI, lost ability to sort columns in datanode tab
Nathan Roberts created HDFS-6407: Summary: new namenode UI, lost ability to sort columns in datanode tab Key: HDFS-6407 URL: https://issues.apache.org/jira/browse/HDFS-6407 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.4.0 Reporter: Nathan Roberts Priority: Minor old ui supported clicking on column header to sort on that column. The new ui seems to have dropped this very useful feature. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6412) Interface audience and stability annotations missing from several new classes related to xattrs.
Chris Nauroth created HDFS-6412:
---
Summary: Interface audience and stability annotations missing from
several new classes related to xattrs.
Key: HDFS-6412
URL: https://issues.apache.org/jira/browse/HDFS-6412
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: HDFS XAttrs (HDFS-2006)
Reporter: Chris Nauroth
Priority: Minor
Attachments: HDFS-6412.1.patch
Let's add the appropriate interface audience and stability annotations to the
following new classes related to xattrs: {{XAttr}}, {{NNConf}} and
{{XAttrHelper}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HDFS-6420) DFSAdmin#refreshNodes should be sent to both NameNodes in HA setup
Jing Zhao created HDFS-6420: --- Summary: DFSAdmin#refreshNodes should be sent to both NameNodes in HA setup Key: HDFS-6420 URL: https://issues.apache.org/jira/browse/HDFS-6420 Project: Hadoop HDFS Issue Type: Bug Reporter: Jing Zhao Assignee: Jing Zhao Currently in HA setup (with logical URI), the DFSAdmin#refreshNodes command is sent to the NameNode first specified in the configuration by default. Users can use -fs option to specify which NN to connect to, but in this case, they usually need to send two separate commands. We should let refreshNodes be sent to both NameNodes by default. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6418) Regression: DFS_NAMENODE_USER_NAME_KEY missing in trunk
Steve Loughran created HDFS-6418:
Summary: Regression: DFS_NAMENODE_USER_NAME_KEY missing in trunk
Key: HDFS-6418
URL: https://issues.apache.org/jira/browse/HDFS-6418
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs-client
Affects Versions: 3.0.0, 2.5.0
Reporter: Steve Loughran
Code i have that compiles against HADOOP 2.4 doesn't build against trunk as
someone took away {{DFSConfigKeys.DFS_NAMENODE_USER_NAME_KEY}} -apparently in
HDFS-6181.
I know the name was obsolete, but anyone who has compiled code using that
reference -rather than cutting and pasting in the string- is going to find
their code doesn't work.
More subtly: that will lead to a link exception trying to run that code on a
2.5+ cluster.
This is a regression: the old names need to go back in, even if they refer to
the new names and are marked as deprecated
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Build failed in Jenkins: Hadoop-Hdfs-trunk #1751
See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1751/changes Changes: [sandy] YARN-1864. Add missing file FSQueueType.java [sandy] YARN-1864. Fair Scheduler Dynamic Hierarchical User Queues (Ashwin Shankar via Sandy Ryza) [arp] HADOOP-10467. Enable proxyuser specification to support list of users in addition to list of groups. (Contributed bt Benoy Antony) [tucu] HADOOP-10556. [FIXING JIRA NUMBER TYPO] Add toLowerCase support to auth_to_local rules for service name. (tucu) [tucu] HADOOP-10566. Add toLowerCase support to auth_to_local rules for service name. (tucu) [wheat9] HADOOP-10471. Reduce the visibility of constants in ProxyUsers. Contributed by Benoy Antony. [cnauroth] HADOOP-10517. InputStream is not closed in two methods of JarFinder. Contributed by Ted Yu. [cnauroth] HADOOP-10541. InputStream in MiniKdc#initKDCServer for minikdc.ldiff is not closed. Contributed by Swarnim Kulkarni. [jeagles] MAPREDUCE-5636. Convert MapReduce Tutorial document to APT (Akira AJISAKA via jeagles) [jeagles] MAPREDUCE-5637. Convert Hadoop Streaming document to APT (Akira AJISAKA via jeagles) [arp] YARN-2018. TestClientRMService.testTokenRenewalWrongUser fails after HADOOP-10562. (Contributed by Ming Ma) [szetszwo] MAPREDUCE-5402. In DynamicInputFormat, change MAX_CHUNKS_TOLERABLE, MAX_CHUNKS_IDEAL, MIN_RECORDS_PER_CHUNK and SPLIT_RATIO to be configurable. Contributed by Tsuyoshi OZAWA [tucu] HADOOP-10433. Key Management Server based on KeyProvider API. (tucu) [umamahesh] HDFS-6337. Setfacl testcase is failing due to dash character in username in TestAclCLI. Contributed by Uma Maheswara Rao G. -- [...truncated 12681 lines...] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.894 sec - in org.apache.hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks Running org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.088 sec - in org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy Running org.apache.hadoop.hdfs.TestClose Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.482 sec - in org.apache.hadoop.hdfs.TestClose Running org.apache.hadoop.hdfs.TestDatanodeDeath Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 140.752 sec - in org.apache.hadoop.hdfs.TestDatanodeDeath Running org.apache.hadoop.hdfs.TestFileAppend Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.618 sec - in org.apache.hadoop.hdfs.TestFileAppend Running org.apache.hadoop.hdfs.TestRollingUpgrade Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 41.076 sec - in org.apache.hadoop.hdfs.TestRollingUpgrade Running org.apache.hadoop.hdfs.TestFileCreationEmpty Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.57 sec - in org.apache.hadoop.hdfs.TestFileCreationEmpty Running org.apache.hadoop.hdfs.TestDatanodeBlockScanner Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 154.055 sec - in org.apache.hadoop.hdfs.TestDatanodeBlockScanner Running org.apache.hadoop.hdfs.TestDFSStorageStateRecovery Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 92.347 sec - in org.apache.hadoop.hdfs.TestDFSStorageStateRecovery Running org.apache.hadoop.hdfs.TestDeprecatedKeys Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.307 sec - in org.apache.hadoop.hdfs.TestDeprecatedKeys Running org.apache.hadoop.hdfs.TestFileAppend2 Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.221 sec - in org.apache.hadoop.hdfs.TestFileAppend2 Running org.apache.hadoop.hdfs.TestDFSUpgrade Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.988 sec - in org.apache.hadoop.hdfs.TestDFSUpgrade Running org.apache.hadoop.hdfs.TestFileCorruption Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.07 sec - in org.apache.hadoop.hdfs.TestFileCorruption Running org.apache.hadoop.hdfs.TestFileAppendRestart Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.2 sec - in org.apache.hadoop.hdfs.TestFileAppendRestart Running org.apache.hadoop.hdfs.TestHDFSFileSystemContract Tests run: 44, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 48.737 sec - in org.apache.hadoop.hdfs.TestHDFSFileSystemContract Running org.apache.hadoop.hdfs.TestQuota Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.377 sec - in org.apache.hadoop.hdfs.TestQuota Running org.apache.hadoop.hdfs.TestDFSAddressConfig Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.049 sec - in org.apache.hadoop.hdfs.TestDFSAddressConfig Running org.apache.hadoop.hdfs.TestAppendDifferentChecksum Tests run: 3, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 8.723 sec - in org.apache.hadoop.hdfs.TestAppendDifferentChecksum Running org.apache.hadoop.hdfs.TestParallelUnixDomainRead Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
[jira] [Created] (HDFS-6411) /hdfs mount raises I/O error and hangs when a unauthorized user attempts to access it
Zhongyi Xie created HDFS-6411: - Summary: /hdfs mount raises I/O error and hangs when a unauthorized user attempts to access it Key: HDFS-6411 URL: https://issues.apache.org/jira/browse/HDFS-6411 Project: Hadoop HDFS Issue Type: Bug Components: nfs Affects Versions: 2.2.0 Reporter: Zhongyi Xie 0) login as root, make sure service nfs-hdfs and nfs-hdfs-client are running: [root@zhongyi-test-cluster-desktop hdfs]# ls /hdfs backups hive mr-history system tmp user 1) add a user nfs-test: adduser nfs-test 2) switch to test user: su - nfs-test 3) access hdfs nfs gateway [nfs-test@zhongyi-test-cluster-desktop ~]$ ls /hdfs ls: cannot open directory /hdfs: Input/output error retry: [nfs-test@zhongyi-test-cluster-desktop ~]$ ls /hdfs ls: cannot access /hdfs: Stale NFS file handle 4) switch back to root and access hdfs nfs gateway [nfs-test@zhongyi-test-cluster-desktop ~]$ exit logout [root@zhongyi-test-cluster-desktop hdfs]# ls /hdfs ls: cannot access /hdfs: Stale NFS file handle the nfsserver log indicates we hit an authorization error in the rpc handler; org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: nfsserver is not allowed to impersonate nfs-test and NFS3ERR_IO is returned, which explains why we see input/output error. One can catch the authorizationexception and return the correct error: NFS3ERR_ACCES to fix the error message on the client side but that doesn't seem to solve the mount hang issue though. When the mount hang happens, it stops printing nfsserver log which makes it more difficult to figure out the real cause of the hang. According to jstack and debugger, the nfsserver seems to be waiting for client requests -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6407) new namenode UI, lost ability to sort columns in datanode tab
Nathan Roberts created HDFS-6407: Summary: new namenode UI, lost ability to sort columns in datanode tab Key: HDFS-6407 URL: https://issues.apache.org/jira/browse/HDFS-6407 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.4.0 Reporter: Nathan Roberts Priority: Minor old ui supported clicking on column header to sort on that column. The new ui seems to have dropped this very useful feature. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6415) Missing null check in FSImageSerialization#writePermissionStatus()
Ted Yu created HDFS-6415:
Summary: Missing null check in
FSImageSerialization#writePermissionStatus()
Key: HDFS-6415
URL: https://issues.apache.org/jira/browse/HDFS-6415
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
PermissionStatus.write(out, inode.getUserName(), inode.getGroupName(), p);
{code}
getUserName() / getGroupName() may return null.
null check should be added for these two calls.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Re: [Vote] Merge The HDFS XAttrs Feature Branch (HDFS-2006) to Trunk
+1 for the merge. I've participated in ongoing design discussions and code reviews on individual patches. Yesterday, I completed a final review pass over the code in the feature branch, the design document, the end user documentation and the test plan. It looks ready to me. Thank you to everyone who made contributions on the feature branch. Chris Nauroth Hortonworks http://hortonworks.com/ On Wed, May 14, 2014 at 9:20 AM, Andrew Wang andrew.w...@cloudera.comwrote: +1 from me as well, I've participated in review and development of this branch and think it's ready for merge. Thanks, Andrew On Wed, May 14, 2014 at 5:47 AM, Gangumalla, Uma uma.ganguma...@intel.comwrote: Hello HDFS Devs, I would like to call for a vote to merge the HDFS Extended Attributes (XAttrs) feature from the HDFS-2006 branch to the trunk. XAttrs are already widely supported on many operating systems, including Linux, Windows, and Mac OS. This will allow storing attributes for HDFS file/directory. XAttr consist of a name and a value and exist in one of 4 namespaces: user, trusted, security, and system. An XAttr name is prefixed with one of these namespaces, so for example, user.myxattr. Consistent with ongoing awareness of Namenode memory usage, the maximum number and size of XAttrs on a file/directory are limited by a configuration parameter. The design document contains more details and can be found here: https://issues.apache.org/jira/secure/attachment/12644341/HDFS-XAttrs-Design-3.pdf Development of this feature has been tracked in JIRA HDFS-2006: https://issues.apache.org/jira/browse/HDFS-2006 All of the development work for the feature is contained in the HDFS-2006 branch: https://svn.apache.org/repos/asf/hadoop/common/branches/HDFS-2006 As last tasks, we are working to support XAttrs via libhdfs, webhdfs as well as other minor improvements. We intend to finish those enhancements before the vote completes and otherwise we could move them to top-level JIRAs as they can be tracked independently. User document is also ready for this feature. Here the doc attached in JIRA: https://issues.apache.org/jira/secure/attachment/12644787/ExtendedAttributes.html The XAttrs feature is backwards-compatible and enabled by default. A cluster administrator can disable it. Testing: We've developed more than 70 new tests which cover the XAttrs get, set and remove APIs through DistributedFileSystem and WebHdfsFileSystem, the new XAttr CLI commands, HA, XAttr persistence in the fsimage and related. Additional testing plans are documented in: https://issues.apache.org/jira/secure/attachment/12644342/Test-Plan-for-Extended-Attributes-1.pdf Thanks a lot to the contributors who have helped and participated in the branch development. Code contributors are Yi Liu, Charles Lamb, Andrew Wang and Uma Maheswara Rao G. The design document incorporates feedback from many community members: Chris Nauroth, Andrew Purtell, Tianyou Li, Avik Dey, Charles Lamb, Alejandro, Andrew Wang, Tsz Wo Nicholas Sze and Uma Maheswara Rao G. Code reviewers on individual patches include Chris Nauroth, Alejandro, Andrew Wang, Charles Lamb, Tsz Wo Nicholas Sze and Uma Maheswara Rao G. Also thanks to Dhruba for bringing up this JIRA and thanks to others who participated for discussions. This vote will run for a week and close on 5/21/2014 at 06:16 pm IST. Here is my +1 to start with. Regards, Uma (umamah...@apache.orgmailto:umamah...@apache.org) -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Resolved] (HDFS-6410) DFSClient unwraps AclException in xattr methods, but those methods cannot throw AclException.
[
https://issues.apache.org/jira/browse/HDFS-6410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang resolved HDFS-6410.
---
Resolution: Fixed
Fix Version/s: HDFS XAttrs (HDFS-2006)
Thanks Yi, committed to branch.
DFSClient unwraps AclException in xattr methods, but those methods cannot
throw AclException.
-
Key: HDFS-6410
URL: https://issues.apache.org/jira/browse/HDFS-6410
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: HDFS XAttrs (HDFS-2006)
Reporter: Chris Nauroth
Assignee: Andrew Wang
Priority: Minor
Fix For: HDFS XAttrs (HDFS-2006)
Attachments: hdfs-6410.1.patch
The various xattr methods in {{DFSClient}} specify {{AclException}} in the
call to {{RemoteException#unwrapRemoteException}}. It's impossible for the
xattr APIs to throw {{AclException}}. Since encountering {{AclException}}
would be an unexpected condition, we should not unwrap it so that instead we
maintain the full stack trace.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HDFS-6419) TestBookKeeperHACheckpoints#TestSBNCheckpoints fails on trunk
Akira AJISAKA created HDFS-6419: --- Summary: TestBookKeeperHACheckpoints#TestSBNCheckpoints fails on trunk Key: HDFS-6419 URL: https://issues.apache.org/jira/browse/HDFS-6419 Project: Hadoop HDFS Issue Type: Test Affects Versions: 2.5.0 Reporter: Akira AJISAKA TestBookKeerHACheckpoints#TestSBNCheckpoints fails on trunk. See https://builds.apache.org/job/PreCommit-HDFS-Build/6908//testReport/ -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6421) RHEL4 fails to compile vecsum.c
Jason Lowe created HDFS-6421: Summary: RHEL4 fails to compile vecsum.c Key: HDFS-6421 URL: https://issues.apache.org/jira/browse/HDFS-6421 Project: Hadoop HDFS Issue Type: Bug Components: libhdfs Affects Versions: 2.5.0 Environment: RHEL4 Reporter: Jason Lowe After HDFS-6287 RHEL4 builds fail trying to compile vecsum.c since they don't have RUSAGE_THREAD. RHEL4 is ancient, but we use it in a 32-bit compatibility environment. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HDFS-6414) xattr modification operations are based on state of latest snapshot instead of current version of inode.
[
https://issues.apache.org/jira/browse/HDFS-6414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth resolved HDFS-6414.
-
Resolution: Fixed
xattr modification operations are based on state of latest snapshot instead
of current version of inode.
Key: HDFS-6414
URL: https://issues.apache.org/jira/browse/HDFS-6414
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: HDFS XAttrs (HDFS-2006)
Reporter: Chris Nauroth
Assignee: Andrew Wang
Fix For: HDFS XAttrs (HDFS-2006)
Attachments: hdfs-6414.1.patch, hdfs-6414.2.patch
{{XAttrStorage#updateINodeXAttrs}} modifies the inode's {{XAttrFeature}}
based on reading its current state. However, the logic for reading current
state is incorrect and may instead read the state of the latest snapshot. If
xattrs have been changed after creation of that snapshot, then subsequent
xattr operations may yield incorrect results.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HDFS-6413) xattr names erroneously handled as case-insensitive.
Chris Nauroth created HDFS-6413: --- Summary: xattr names erroneously handled as case-insensitive. Key: HDFS-6413 URL: https://issues.apache.org/jira/browse/HDFS-6413 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: HDFS XAttrs (HDFS-2006) Reporter: Chris Nauroth Attachments: HDFS-6413.1.patch Xattr names currently are handled as case-insensitive. The names should be case-sensitive instead. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6416) Use Time#monotonicNow in OpenFileCtx and OpenFileCtxCatch to avoid system clock bugs
Brandon Li created HDFS-6416: Summary: Use Time#monotonicNow in OpenFileCtx and OpenFileCtxCatch to avoid system clock bugs Key: HDFS-6416 URL: https://issues.apache.org/jira/browse/HDFS-6416 Project: Hadoop HDFS Issue Type: Improvement Components: nfs Reporter: Brandon Li Priority: Minor As [~cnauroth] pointed out in HADOOP-10612, Time#monotonicNow is a more preferred method to use since this isn't subject to system clock bugs (i.e. Someone resets the clock to a time in the past, and then updates don't happen for a long time.) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HDFS-6314) Test cases for XAttrs
[ https://issues.apache.org/jira/browse/HDFS-6314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G resolved HDFS-6314. --- Resolution: Fixed Hadoop Flags: Reviewed I have just committed this to branch! Test cases for XAttrs - Key: HDFS-6314 URL: https://issues.apache.org/jira/browse/HDFS-6314 Project: Hadoop HDFS Issue Type: Sub-task Components: test Affects Versions: HDFS XAttrs (HDFS-2006) Reporter: Yi Liu Assignee: Yi Liu Fix For: HDFS XAttrs (HDFS-2006) Attachments: HDFS-6314.1.patch, HDFS-6314.2.patch, HDFS-6314.3.patch, HDFS-6314.4.patch, HDFS-6314.patch Tests NameNode interaction for all XAttr APIs, covers restarting NN, saving new checkpoint. Tests XAttr for Snapshot, symlinks. Tests XAttr for HA failover. And more... -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6410) DFSClient unwraps AclException in xattr methods, but those methods cannot throw AclException.
Chris Nauroth created HDFS-6410:
---
Summary: DFSClient unwraps AclException in xattr methods, but
those methods cannot throw AclException.
Key: HDFS-6410
URL: https://issues.apache.org/jira/browse/HDFS-6410
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: HDFS XAttrs (HDFS-2006)
Reporter: Chris Nauroth
Priority: Minor
Attachments: hdfs-6410.1.patch
The various xattr methods in {{DFSClient}} specify {{AclException}} in the call
to {{RemoteException#unwrapRemoteException}}. It's impossible for the xattr
APIs to throw {{AclException}}. Since encountering {{AclException}} would be
an unexpected condition, we should not unwrap it so that instead we maintain
the full stack trace.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HDFS-6422) getfattr in CLI doesn't throw exception or return non-0 return code when xattr doesn't exist
Charles Lamb created HDFS-6422: -- Summary: getfattr in CLI doesn't throw exception or return non-0 return code when xattr doesn't exist Key: HDFS-6422 URL: https://issues.apache.org/jira/browse/HDFS-6422 Project: Hadoop HDFS Issue Type: Bug Reporter: Charles Lamb If you do hdfs dfs -getfattr -n user.blah /foo and user.blah doesn't exist, the command prints # file: /foo and a 0 return code. It should print an exception and return a non-0 return code instead. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: [Vote] Merge The HDFS XAttrs Feature Branch (HDFS-2006) to Trunk
I have not looked at the development closely. With rolling upgrades feature support in, are there any incompatible changes with this feature? Sent from phone On May 16, 2014, at 10:30 AM, Chris Nauroth cnaur...@hortonworks.com wrote: +1 for the merge. I've participated in ongoing design discussions and code reviews on individual patches. Yesterday, I completed a final review pass over the code in the feature branch, the design document, the end user documentation and the test plan. It looks ready to me. Thank you to everyone who made contributions on the feature branch. Chris Nauroth Hortonworks http://hortonworks.com/ On Wed, May 14, 2014 at 9:20 AM, Andrew Wang andrew.w...@cloudera.comwrote: +1 from me as well, I've participated in review and development of this branch and think it's ready for merge. Thanks, Andrew On Wed, May 14, 2014 at 5:47 AM, Gangumalla, Uma uma.ganguma...@intel.comwrote: Hello HDFS Devs, I would like to call for a vote to merge the HDFS Extended Attributes (XAttrs) feature from the HDFS-2006 branch to the trunk. XAttrs are already widely supported on many operating systems, including Linux, Windows, and Mac OS. This will allow storing attributes for HDFS file/directory. XAttr consist of a name and a value and exist in one of 4 namespaces: user, trusted, security, and system. An XAttr name is prefixed with one of these namespaces, so for example, user.myxattr. Consistent with ongoing awareness of Namenode memory usage, the maximum number and size of XAttrs on a file/directory are limited by a configuration parameter. The design document contains more details and can be found here: https://issues.apache.org/jira/secure/attachment/12644341/HDFS-XAttrs-Design-3.pdf Development of this feature has been tracked in JIRA HDFS-2006: https://issues.apache.org/jira/browse/HDFS-2006 All of the development work for the feature is contained in the HDFS-2006 branch: https://svn.apache.org/repos/asf/hadoop/common/branches/HDFS-2006 As last tasks, we are working to support XAttrs via libhdfs, webhdfs as well as other minor improvements. We intend to finish those enhancements before the vote completes and otherwise we could move them to top-level JIRAs as they can be tracked independently. User document is also ready for this feature. Here the doc attached in JIRA: https://issues.apache.org/jira/secure/attachment/12644787/ExtendedAttributes.html The XAttrs feature is backwards-compatible and enabled by default. A cluster administrator can disable it. Testing: We've developed more than 70 new tests which cover the XAttrs get, set and remove APIs through DistributedFileSystem and WebHdfsFileSystem, the new XAttr CLI commands, HA, XAttr persistence in the fsimage and related. Additional testing plans are documented in: https://issues.apache.org/jira/secure/attachment/12644342/Test-Plan-for-Extended-Attributes-1.pdf Thanks a lot to the contributors who have helped and participated in the branch development. Code contributors are Yi Liu, Charles Lamb, Andrew Wang and Uma Maheswara Rao G. The design document incorporates feedback from many community members: Chris Nauroth, Andrew Purtell, Tianyou Li, Avik Dey, Charles Lamb, Alejandro, Andrew Wang, Tsz Wo Nicholas Sze and Uma Maheswara Rao G. Code reviewers on individual patches include Chris Nauroth, Alejandro, Andrew Wang, Charles Lamb, Tsz Wo Nicholas Sze and Uma Maheswara Rao G. Also thanks to Dhruba for bringing up this JIRA and thanks to others who participated for discussions. This vote will run for a week and close on 5/21/2014 at 06:16 pm IST. Here is my +1 to start with. Regards, Uma (umamah...@apache.orgmailto:umamah...@apache.org) -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Resolved] (HDFS-6412) Interface audience and stability annotations missing from several new classes related to xattrs.
[
https://issues.apache.org/jira/browse/HDFS-6412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang resolved HDFS-6412.
---
Resolution: Fixed
Fix Version/s: HDFS XAttrs (HDFS-2006)
Thanks Yi, committed to branch.
Interface audience and stability annotations missing from several new classes
related to xattrs.
Key: HDFS-6412
URL: https://issues.apache.org/jira/browse/HDFS-6412
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: HDFS XAttrs (HDFS-2006)
Reporter: Chris Nauroth
Assignee: Andrew Wang
Priority: Minor
Fix For: HDFS XAttrs (HDFS-2006)
Attachments: HDFS-6412.1.patch
Let's add the appropriate interface audience and stability annotations to the
following new classes related to xattrs: {{XAttr}}, {{NNConf}} and
{{XAttrHelper}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HDFS-6409) Fix typo in log message about NameNode layout version upgrade.
Chris Nauroth created HDFS-6409: --- Summary: Fix typo in log message about NameNode layout version upgrade. Key: HDFS-6409 URL: https://issues.apache.org/jira/browse/HDFS-6409 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.4.0, 3.0.0 Reporter: Chris Nauroth Priority: Trivial During startup, the NameNode logs a message if the existing metadata is using an old layout version. This message contains a minor typo. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6414) xattr modification operations are based on state of latest snapshot instead of current version of inode.
Chris Nauroth created HDFS-6414:
---
Summary: xattr modification operations are based on state of
latest snapshot instead of current version of inode.
Key: HDFS-6414
URL: https://issues.apache.org/jira/browse/HDFS-6414
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: HDFS XAttrs (HDFS-2006)
Reporter: Chris Nauroth
Attachments: hdfs-6414.1.patch, hdfs-6414.2.patch
{{XAttrStorage#updateINodeXAttrs}} modifies the inode's {{XAttrFeature}} based
on reading its current state. However, the logic for reading current state is
incorrect and may instead read the state of the latest snapshot. If xattrs
have been changed after creation of that snapshot, then subsequent xattr
operations may yield incorrect results.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Re: [Vote] Merge The HDFS XAttrs Feature Branch (HDFS-2006) to Trunk
Hi Suresh, HDFS-2006 adds new edit log ops (no modifications to existing ops) and a new XAttr INode Feature needs to be serialized to the fsimage. This required doing a NN layout version bump, so a 2.4 NN will not be able to start with these edit logs or images. Thus it's incompatible. I think you're better equipped to comment on rolling upgrades, but based on my understanding, I think the above means we should still be able to rolling upgrade between 2.4 and whatever version eventually includes xattr support. RPC-wise, HDFS-2006 only adds new RPCs, so I don't think there are any concerns there. Thanks, Andrew On Fri, May 16, 2014 at 2:56 PM, Suresh Srinivas sur...@hortonworks.comwrote: I have not looked at the development closely. With rolling upgrades feature support in, are there any incompatible changes with this feature? Sent from phone On May 16, 2014, at 10:30 AM, Chris Nauroth cnaur...@hortonworks.com wrote: +1 for the merge. I've participated in ongoing design discussions and code reviews on individual patches. Yesterday, I completed a final review pass over the code in the feature branch, the design document, the end user documentation and the test plan. It looks ready to me. Thank you to everyone who made contributions on the feature branch. Chris Nauroth Hortonworks http://hortonworks.com/ On Wed, May 14, 2014 at 9:20 AM, Andrew Wang andrew.w...@cloudera.com wrote: +1 from me as well, I've participated in review and development of this branch and think it's ready for merge. Thanks, Andrew On Wed, May 14, 2014 at 5:47 AM, Gangumalla, Uma uma.ganguma...@intel.comwrote: Hello HDFS Devs, I would like to call for a vote to merge the HDFS Extended Attributes (XAttrs) feature from the HDFS-2006 branch to the trunk. XAttrs are already widely supported on many operating systems, including Linux, Windows, and Mac OS. This will allow storing attributes for HDFS file/directory. XAttr consist of a name and a value and exist in one of 4 namespaces: user, trusted, security, and system. An XAttr name is prefixed with one of these namespaces, so for example, user.myxattr. Consistent with ongoing awareness of Namenode memory usage, the maximum number and size of XAttrs on a file/directory are limited by a configuration parameter. The design document contains more details and can be found here: https://issues.apache.org/jira/secure/attachment/12644341/HDFS-XAttrs-Design-3.pdf Development of this feature has been tracked in JIRA HDFS-2006: https://issues.apache.org/jira/browse/HDFS-2006 All of the development work for the feature is contained in the HDFS-2006 branch: https://svn.apache.org/repos/asf/hadoop/common/branches/HDFS-2006 As last tasks, we are working to support XAttrs via libhdfs, webhdfs as well as other minor improvements. We intend to finish those enhancements before the vote completes and otherwise we could move them to top-level JIRAs as they can be tracked independently. User document is also ready for this feature. Here the doc attached in JIRA: https://issues.apache.org/jira/secure/attachment/12644787/ExtendedAttributes.html The XAttrs feature is backwards-compatible and enabled by default. A cluster administrator can disable it. Testing: We've developed more than 70 new tests which cover the XAttrs get, set and remove APIs through DistributedFileSystem and WebHdfsFileSystem, the new XAttr CLI commands, HA, XAttr persistence in the fsimage and related. Additional testing plans are documented in: https://issues.apache.org/jira/secure/attachment/12644342/Test-Plan-for-Extended-Attributes-1.pdf Thanks a lot to the contributors who have helped and participated in the branch development. Code contributors are Yi Liu, Charles Lamb, Andrew Wang and Uma Maheswara Rao G. The design document incorporates feedback from many community members: Chris Nauroth, Andrew Purtell, Tianyou Li, Avik Dey, Charles Lamb, Alejandro, Andrew Wang, Tsz Wo Nicholas Sze and Uma Maheswara Rao G. Code reviewers on individual patches include Chris Nauroth, Alejandro, Andrew Wang, Charles Lamb, Tsz Wo Nicholas Sze and Uma Maheswara Rao G. Also thanks to Dhruba for bringing up this JIRA and thanks to others who participated for discussions. This vote will run for a week and close on 5/21/2014 at 06:16 pm IST. Here is my +1 to start with. Regards, Uma (umamah...@apache.orgmailto:umamah...@apache.org) -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this
[jira] [Resolved] (HDFS-5621) NameNode: add indicator in web UI file system browser if a file has an ACL.
[ https://issues.apache.org/jira/browse/HDFS-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth resolved HDFS-5621. - Resolution: Duplicate I'm resolving this as duplicate of HDFS-6326. I ended up implementing this within the scope of that patch, so the indicator will be there in the 2.4.1 release. NameNode: add indicator in web UI file system browser if a file has an ACL. --- Key: HDFS-5621 URL: https://issues.apache.org/jira/browse/HDFS-5621 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 2.4.0 Reporter: Chris Nauroth Assignee: Haohui Mai Attachments: HDFS-5621.000.patch Change the file system browser to append the '+' character to permissions of any file or directory that has an ACL. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6424) blockReport doesn't need to invalidate blocks on SBN
Ming Ma created HDFS-6424: - Summary: blockReport doesn't need to invalidate blocks on SBN Key: HDFS-6424 URL: https://issues.apache.org/jira/browse/HDFS-6424 Project: Hadoop HDFS Issue Type: Bug Reporter: Ming Ma Assignee: Ming Ma After the fix in https://issues.apache.org/jira/browse/HDFS-6178, blockManager no longer compute pending replication work on SBN. As part of that, it also stop removing items from invalidateBlocks. blocks can still be added to invalidateBlocks on SBN as part of blockReport. As a result, the PendingDeletionBlocks metrics will keep going up on SBN. To fix that, we don't need to add blocks to invalidateBlocks during blockReport for SBN. -- This message was sent by Atlassian JIRA (v6.2#6252)
