[jira] [Commented] (HADOOP-8817) Backport Network Topology Extension for Virtualization (HADOOP-8468) to branch-1
[ https://issues.apache.org/jira/browse/HADOOP-8817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13456889#comment-13456889 ] Junping Du commented on HADOOP-8817: Thanks Suresh for the correction here. So branch-1.2 is next feature branch (1.1 is next bug-fix branch). Isn't it? Backport Network Topology Extension for Virtualization (HADOOP-8468) to branch-1 Key: HADOOP-8817 URL: https://issues.apache.org/jira/browse/HADOOP-8817 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 1.0.0 Reporter: Junping Du Assignee: Junping Du Labels: features Attachments: HADOOP-8817.patch HADOOP-8468 propose network topology changes for running on virtualized infrastructure, which includes: 1. Add NodeGroup layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup): HADOOP-8469, HADOOP-8470 2. Update Replica Placement/Removal Policy to reflect new topology layer: HDFS-3498, HDFS-3601 3. Update balancer policy:HDFS-3495 4. Update Task Scheduling Policy to reflect new topology layer and support the case that compute nodes (NodeManager or TaskTracker) and data nodes are separated into different VMs, but still benefit from physical host locality: YARN-18, YARN-19. This JIRA will address the backport work on branch-1 which will be divided into 4 issues/patches in related jira issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8468) Umbrella of enhancements to support different failure and locality topologies
[ https://issues.apache.org/jira/browse/HADOOP-8468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13456891#comment-13456891 ] Junping Du commented on HADOOP-8468: Thanks for great comments. Konstantin. the doc revised-1.0 already address full policy definition. Hi guys, I am back porting patches to branch-1. Hope I can get your support and help on reviewing. :) Umbrella of enhancements to support different failure and locality topologies - Key: HADOOP-8468 URL: https://issues.apache.org/jira/browse/HADOOP-8468 Project: Hadoop Common Issue Type: Improvement Components: ha, io Affects Versions: 1.0.0, 2.0.0-alpha Reporter: Junping Du Assignee: Junping Du Attachments: HADOOP-8468-total.patch, HADOOP-8468-total-v3.patch, Proposal for enchanced failure and locality topologies.pdf, Proposal for enchanced failure and locality topologies (revised-1.0).pdf The current hadoop network topology (described in some previous issues like: Hadoop-692) works well in classic three-tiers network when it comes out. However, it does not take into account other failure models or changes in the infrastructure that can affect network bandwidth efficiency like: virtualization. Virtualized platform has following genes that shouldn't been ignored by hadoop topology in scheduling tasks, placing replica, do balancing or fetching block for reading: 1. VMs on the same physical host are affected by the same hardware failure. In order to match the reliability of a physical deployment, replication of data across two virtual machines on the same host should be avoided. 2. The network between VMs on the same physical host has higher throughput and lower latency and does not consume any physical switch bandwidth. Thus, we propose to make hadoop network topology extend-able and introduce a new level in the hierarchical topology, a node group level, which maps well onto an infrastructure that is based on a virtualized environment. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8818) Should use equals() rather than == to compare String or Text in MD5MD5CRC32FileChecksum and TFileDumper
[ https://issues.apache.org/jira/browse/HADOOP-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13456980#comment-13456980 ] Hudson commented on HADOOP-8818: Integrated in Hadoop-Hdfs-trunk #1168 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1168/]) HADOOP-8818. Use equals instead == in MD5MD5CRC32FileChecksum and TFileDumper. Contributed by Brandon Li. (Revision 1385374) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1385374 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/MD5MD5CRC32FileChecksum.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/TFileDumper.java Should use equals() rather than == to compare String or Text in MD5MD5CRC32FileChecksum and TFileDumper --- Key: HADOOP-8818 URL: https://issues.apache.org/jira/browse/HADOOP-8818 Project: Hadoop Common Issue Type: Bug Components: fs, io Affects Versions: 3.0.0 Reporter: Brandon Li Assignee: Brandon Li Priority: Minor Fix For: 3.0.0 Attachments: HADOOP-8818.patch Should use equals() rather than == to compare String or Text in MD5MD5CRC32FileChecksum and TFileDumper. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8821) Findbugs warning Configuration.dumpDeprecatedKeys() concatenates strings using + in a loop
[ https://issues.apache.org/jira/browse/HADOOP-8821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13456981#comment-13456981 ] Hudson commented on HADOOP-8821: Integrated in Hadoop-Hdfs-trunk #1168 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1168/]) HADOOP-8821. Fix findbugs warning related concatenating string in a for loop in Configuration#dumpDeprecatedKeys(). Contributed by Suresh Srinivas. (Revision 1385389) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1385389 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java Findbugs warning Configuration.dumpDeprecatedKeys() concatenates strings using + in a loop -- Key: HADOOP-8821 URL: https://issues.apache.org/jira/browse/HADOOP-8821 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Reporter: Suresh Srinivas Assignee: Suresh Srinivas Priority: Trivial Fix For: 3.0.0 Attachments: HADOOP-8821.patch Configuration.dumpDeprecatedKeys() concatenates strings using + in a loop. Instead use StringBuilder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8819) Should use instead of in a few places in FTPFileSystem,FTPInputStream,S3InputStream,ViewFileSystem,ViewFs
[ https://issues.apache.org/jira/browse/HADOOP-8819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13456982#comment-13456982 ] Hudson commented on HADOOP-8819: Integrated in Hadoop-Hdfs-trunk #1168 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1168/]) HADOOP-8819. Incorrectly is used instead of in some file system implementations. Contributed by Brandon Li. (Revision 1386451) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1386451 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ftp/FTPFileSystem.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ftp/FTPInputStream.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3/S3InputStream.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java Should use instead of in a few places in FTPFileSystem,FTPInputStream,S3InputStream,ViewFileSystem,ViewFs --- Key: HADOOP-8819 URL: https://issues.apache.org/jira/browse/HADOOP-8819 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Brandon Li Assignee: Brandon Li Fix For: 3.0.0, 2.0.3-alpha Attachments: HADOOP-8819.patch Should use instead of in a few places in FTPFileSystem,FTPInputStream,S3InputStream,ViewFileSystem,ViewFs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8819) Should use instead of in a few places in FTPFileSystem,FTPInputStream,S3InputStream,ViewFileSystem,ViewFs
[ https://issues.apache.org/jira/browse/HADOOP-8819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13457032#comment-13457032 ] Hudson commented on HADOOP-8819: Integrated in Hadoop-Mapreduce-trunk #1199 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1199/]) HADOOP-8819. Incorrectly is used instead of in some file system implementations. Contributed by Brandon Li. (Revision 1386451) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1386451 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ftp/FTPFileSystem.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ftp/FTPInputStream.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3/S3InputStream.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java Should use instead of in a few places in FTPFileSystem,FTPInputStream,S3InputStream,ViewFileSystem,ViewFs --- Key: HADOOP-8819 URL: https://issues.apache.org/jira/browse/HADOOP-8819 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Brandon Li Assignee: Brandon Li Fix For: 3.0.0, 2.0.3-alpha Attachments: HADOOP-8819.patch Should use instead of in a few places in FTPFileSystem,FTPInputStream,S3InputStream,ViewFileSystem,ViewFs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8821) Findbugs warning Configuration.dumpDeprecatedKeys() concatenates strings using + in a loop
[ https://issues.apache.org/jira/browse/HADOOP-8821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13457031#comment-13457031 ] Hudson commented on HADOOP-8821: Integrated in Hadoop-Mapreduce-trunk #1199 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1199/]) HADOOP-8821. Fix findbugs warning related concatenating string in a for loop in Configuration#dumpDeprecatedKeys(). Contributed by Suresh Srinivas. (Revision 1385389) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1385389 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java Findbugs warning Configuration.dumpDeprecatedKeys() concatenates strings using + in a loop -- Key: HADOOP-8821 URL: https://issues.apache.org/jira/browse/HADOOP-8821 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Reporter: Suresh Srinivas Assignee: Suresh Srinivas Priority: Trivial Fix For: 3.0.0 Attachments: HADOOP-8821.patch Configuration.dumpDeprecatedKeys() concatenates strings using + in a loop. Instead use StringBuilder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8811) Compile hadoop native library in FreeBSD
[ https://issues.apache.org/jira/browse/HADOOP-8811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13457092#comment-13457092 ] Hadoop QA commented on HADOOP-8811: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12545425/freebsd-native-flags-hardlink.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.ha.TestZKFailoverController +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1480//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1480//console This message is automatically generated. Compile hadoop native library in FreeBSD Key: HADOOP-8811 URL: https://issues.apache.org/jira/browse/HADOOP-8811 Project: Hadoop Common Issue Type: Bug Components: native Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0 Reporter: Radim Kolar Priority: Critical Labels: freebsd Attachments: freebsd-native-flags-hardlink.txt, freebsd-native-flags.txt, freebsd-native-trunk.txt, freebsd-native.txt Native hadoop library do not compiles in FreeBSD because setnetgrent returns void and assembler do not supports SSE4 instructions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8816) HTTP Error 413 full HEAD if using kerberos authentication
[ https://issues.apache.org/jira/browse/HADOOP-8816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13457119#comment-13457119 ] Alejandro Abdelnur commented on HADOOP-8816: 128K as header buffer size seems a bit too big. Could this be related to this? http://www.novell.com/communities/node/11516/kerberos-authentication-may-fail-access-manager-identity-server-users-large-group-members Would be possible to get the actual header size that is making things to fail? HTTP Error 413 full HEAD if using kerberos authentication - Key: HADOOP-8816 URL: https://issues.apache.org/jira/browse/HADOOP-8816 Project: Hadoop Common Issue Type: Bug Components: net Affects Versions: 2.0.1-alpha Environment: ubuntu linux with active directory kerberos. Reporter: Moritz Moeller The HTTP Authentication: header is too large if using kerberos and the request is rejected by Jetty because Jetty has a too low default header size limit. Can be fixed by adding ret.setHeaderBufferSize(1024*128); in org.apache.hadoop.http.HttpServer.createDefaultChannelConnector -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8816) HTTP Error 413 full HEAD if using kerberos authentication
[ https://issues.apache.org/jira/browse/HADOOP-8816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13457121#comment-13457121 ] Alejandro Abdelnur commented on HADOOP-8816: Also, if we tweak the header buffer size, we should doing it in a configurable way. HTTP Error 413 full HEAD if using kerberos authentication - Key: HADOOP-8816 URL: https://issues.apache.org/jira/browse/HADOOP-8816 Project: Hadoop Common Issue Type: Bug Components: net Affects Versions: 2.0.1-alpha Environment: ubuntu linux with active directory kerberos. Reporter: Moritz Moeller The HTTP Authentication: header is too large if using kerberos and the request is rejected by Jetty because Jetty has a too low default header size limit. Can be fixed by adding ret.setHeaderBufferSize(1024*128); in org.apache.hadoop.http.HttpServer.createDefaultChannelConnector -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8816) HTTP Error 413 full HEAD if using kerberos authentication
[ https://issues.apache.org/jira/browse/HADOOP-8816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13457129#comment-13457129 ] Moritz Moeller commented on HADOOP-8816: No, Kerberos tokens do not contain group membership information, but tend to get pretty large, 4-8k base64 encoded. I guess 16kb header size would be enough. Making that configurable is your choice, I personally wouldn't as I know no things that cause header sizes larger than Kerberos, but then if it was configurable already this ticket wouldn't exist. HTTP Error 413 full HEAD if using kerberos authentication - Key: HADOOP-8816 URL: https://issues.apache.org/jira/browse/HADOOP-8816 Project: Hadoop Common Issue Type: Bug Components: net Affects Versions: 2.0.1-alpha Environment: ubuntu linux with active directory kerberos. Reporter: Moritz Moeller The HTTP Authentication: header is too large if using kerberos and the request is rejected by Jetty because Jetty has a too low default header size limit. Can be fixed by adding ret.setHeaderBufferSize(1024*128); in org.apache.hadoop.http.HttpServer.createDefaultChannelConnector -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8819) Should use instead of in a few places in FTPFileSystem,FTPInputStream,S3InputStream,ViewFileSystem,ViewFs
[ https://issues.apache.org/jira/browse/HADOOP-8819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HADOOP-8819: --- Attachment: HADOOP-8819.branch-1.patch Should use instead of in a few places in FTPFileSystem,FTPInputStream,S3InputStream,ViewFileSystem,ViewFs --- Key: HADOOP-8819 URL: https://issues.apache.org/jira/browse/HADOOP-8819 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Brandon Li Assignee: Brandon Li Fix For: 3.0.0, 2.0.3-alpha Attachments: HADOOP-8819.branch-1.patch, HADOOP-8819.patch Should use instead of in a few places in FTPFileSystem,FTPInputStream,S3InputStream,ViewFileSystem,ViewFs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8819) Should use instead of in a few places in FTPFileSystem,FTPInputStream,S3InputStream,ViewFileSystem,ViewFs
[ https://issues.apache.org/jira/browse/HADOOP-8819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13457148#comment-13457148 ] Brandon Li commented on HADOOP-8819: The same problem exists in branch-1 (except viewfs). Uploaded a branch-1 patch. Removed created this time. :-) Should use instead of in a few places in FTPFileSystem,FTPInputStream,S3InputStream,ViewFileSystem,ViewFs --- Key: HADOOP-8819 URL: https://issues.apache.org/jira/browse/HADOOP-8819 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Brandon Li Assignee: Brandon Li Fix For: 3.0.0, 2.0.3-alpha Attachments: HADOOP-8819.branch-1.patch, HADOOP-8819.patch Should use instead of in a few places in FTPFileSystem,FTPInputStream,S3InputStream,ViewFileSystem,ViewFs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8819) Should use instead of in a few places in FTPFileSystem,FTPInputStream,S3InputStream,ViewFileSystem,ViewFs
[ https://issues.apache.org/jira/browse/HADOOP-8819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13457150#comment-13457150 ] Hadoop QA commented on HADOOP-8819: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12545438/HADOOP-8819.branch-1.patch against trunk revision . -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1481//console This message is automatically generated. Should use instead of in a few places in FTPFileSystem,FTPInputStream,S3InputStream,ViewFileSystem,ViewFs --- Key: HADOOP-8819 URL: https://issues.apache.org/jira/browse/HADOOP-8819 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Brandon Li Assignee: Brandon Li Fix For: 3.0.0, 2.0.3-alpha Attachments: HADOOP-8819.branch-1.patch, HADOOP-8819.patch Should use instead of in a few places in FTPFileSystem,FTPInputStream,S3InputStream,ViewFileSystem,ViewFs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8733) TestStreamingTaskLog, TestJvmManager, TestLinuxTaskControllerLaunchArgs fail on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13457152#comment-13457152 ] Vinod Kumar Vavilapalli commented on HADOOP-8733: - TestJVMManager is simulating the kill code-path in case of linux. So yes, unless it is very difficult, please change the test to verify that the attemptid that is sent across can be successfully used for killing processes in case of Windows. TestStreamingTaskLog, TestJvmManager, TestLinuxTaskControllerLaunchArgs fail on Windows --- Key: HADOOP-8733 URL: https://issues.apache.org/jira/browse/HADOOP-8733 Project: Hadoop Common Issue Type: Bug Components: test Affects Versions: 1-win Reporter: Ivan Mitic Assignee: Ivan Mitic Attachments: HADOOP-8733-scripts.2.patch, HADOOP-8733-scripts.2.patch, HADOOP-8733-scripts.patch Jira tracking test failures related to test .sh script dependencies. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8819) Should use instead of in a few places in FTPFileSystem,FTPInputStream,S3InputStream,ViewFileSystem,ViewFs
[ https://issues.apache.org/jira/browse/HADOOP-8819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HADOOP-8819: Resolution: Fixed Fix Version/s: 1.2.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) +1 for the the branch-1 patch. Thank you Brandon! Should use instead of in a few places in FTPFileSystem,FTPInputStream,S3InputStream,ViewFileSystem,ViewFs --- Key: HADOOP-8819 URL: https://issues.apache.org/jira/browse/HADOOP-8819 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Brandon Li Assignee: Brandon Li Fix For: 1.2.0, 3.0.0, 2.0.3-alpha Attachments: HADOOP-8819.branch-1.patch, HADOOP-8819.patch Should use instead of in a few places in FTPFileSystem,FTPInputStream,S3InputStream,ViewFileSystem,ViewFs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8615) EOFException in DecompressorStream.java needs to be more verbose
[
https://issues.apache.org/jira/browse/HADOOP-8615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13457199#comment-13457199
]
Andy Isaacson commented on HADOOP-8615:
---
Thomas,
Thank you for the patch!
bq. Please let me know if there is any procedure for making this tested only on
hadoop common
release 0.20.2
Don't worry about the robots testing against the wrong branch, you're not doing
anything wrong.
It seems to me this change would be a good thing on trunk as well. Can you port
the patch to trunk?
bq. When the user uses this method and pass the filename, it would be printed
in the EOF exception thrown, if any. So I believe the test cases may not be
necessary. I was able to test it locally by forcefully creating an EOF
Exception and verifying the new message as java.io.EOFException: Unexpected
end of input stream in the file = filename
I think this should be fairly easy to test -- just write a compressed stream,
truncate the compressed stream, then try to read it, catch the EOFException and
verify that the filename shows up in the exception text. Or am I missing
something?
I'm a little worried about the places where your {{fileName}}-using methods add
new default values, for example:
{code}
+ public CompressionInputStream createInputStream(InputStream in,
+Decompressor decompressor, String fileName)
+ throws IOException {
+return new DecompressorStream(in, decompressor,
+ conf.getInt(io.file.buffer.size, 4*1024),fileName);
+ }
{code}
I'll have to think about it longer, but having a default value of 4k hidden in
this method seems wrong to me at a first glance. There are a few other
instances of this as well.
EOFException in DecompressorStream.java needs to be more verbose
Key: HADOOP-8615
URL: https://issues.apache.org/jira/browse/HADOOP-8615
Project: Hadoop Common
Issue Type: Bug
Components: io
Affects Versions: 0.20.2
Reporter: Jeff Lord
Labels: patch
Attachments: HADOOP-8615-release-0.20.2.patch
In ./src/core/org/apache/hadoop/io/compress/DecompressorStream.java
The following exception should at least pass back the file that it encounters
this error in relation to:
protected void getCompressedData() throws IOException {
checkStream();
int n = in.read(buffer, 0, buffer.length);
if (n == -1) {
throw new EOFException(Unexpected end of input stream);
}
This would help greatly to debug bad/corrupt files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HADOOP-7688) When a servlet filter throws an exception in init(..), the Jetty server failed silently.
[ https://issues.apache.org/jira/browse/HADOOP-7688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G reopened HADOOP-7688: - When a servlet filter throws an exception in init(..), the Jetty server failed silently. - Key: HADOOP-7688 URL: https://issues.apache.org/jira/browse/HADOOP-7688 Project: Hadoop Common Issue Type: Improvement Affects Versions: 0.23.0, 0.24.0 Reporter: Tsz Wo (Nicholas), SZE Assignee: Uma Maheswara Rao G Fix For: 3.0.0, 2.0.3-alpha Attachments: filter-init-exception-test.patch, HADOOP-7688-branch-1.patch, HADOOP-7688-branch-2.patch, HADOOP-7688.patch, org.apache.hadoop.http.TestServletFilter-output.txt When a servlet filter throws a ServletException in init(..), the exception is logged by Jetty but not re-throws to the caller. As a result, the Jetty server failed silently. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7688) When a servlet filter throws an exception in init(..), the Jetty server failed silently.
[ https://issues.apache.org/jira/browse/HADOOP-7688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HADOOP-7688: Attachment: HADOOP-7688-branch-1.patch I have just committed into branch-1 When a servlet filter throws an exception in init(..), the Jetty server failed silently. - Key: HADOOP-7688 URL: https://issues.apache.org/jira/browse/HADOOP-7688 Project: Hadoop Common Issue Type: Improvement Affects Versions: 0.23.0, 0.24.0 Reporter: Tsz Wo (Nicholas), SZE Assignee: Uma Maheswara Rao G Fix For: 3.0.0, 2.0.3-alpha Attachments: filter-init-exception-test.patch, HADOOP-7688-branch-1.patch, HADOOP-7688-branch-2.patch, HADOOP-7688.patch, org.apache.hadoop.http.TestServletFilter-output.txt When a servlet filter throws a ServletException in init(..), the exception is logged by Jetty but not re-throws to the caller. As a result, the Jetty server failed silently. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-7688) When a servlet filter throws an exception in init(..), the Jetty server failed silently.
[ https://issues.apache.org/jira/browse/HADOOP-7688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G resolved HADOOP-7688. - Resolution: Fixed Fix Version/s: 1.2.0 When a servlet filter throws an exception in init(..), the Jetty server failed silently. - Key: HADOOP-7688 URL: https://issues.apache.org/jira/browse/HADOOP-7688 Project: Hadoop Common Issue Type: Improvement Affects Versions: 0.23.0, 0.24.0 Reporter: Tsz Wo (Nicholas), SZE Assignee: Uma Maheswara Rao G Fix For: 1.2.0, 3.0.0, 2.0.3-alpha Attachments: filter-init-exception-test.patch, HADOOP-7688-branch-1.patch, HADOOP-7688-branch-2.patch, HADOOP-7688.patch, org.apache.hadoop.http.TestServletFilter-output.txt When a servlet filter throws a ServletException in init(..), the exception is logged by Jetty but not re-throws to the caller. As a result, the Jetty server failed silently. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-8804) Improve Web UIs when the wildcard address is used
[ https://issues.apache.org/jira/browse/HADOOP-8804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins reassigned HADOOP-8804: --- Assignee: Vinay Improve Web UIs when the wildcard address is used - Key: HADOOP-8804 URL: https://issues.apache.org/jira/browse/HADOOP-8804 Project: Hadoop Common Issue Type: Improvement Affects Versions: 1.0.0, 2.0.0-alpha Reporter: Eli Collins Assignee: Vinay Priority: Minor Labels: newbie Attachments: HADOOP-8804-1.0.patch, HADOOP-8804-trunk.patch When IPC addresses are bound to the wildcard (ie the default config) the NN, JT (and probably RM etc) Web UIs are a little goofy. Eg 0 Hadoop Map/Reduce Administration and NameNode '0.0.0.0:18021' (active). Let's improve them. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8805) Move protocol buffer implementation of GetUserMappingProtocol from HDFS to Common
[ https://issues.apache.org/jira/browse/HADOOP-8805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13457235#comment-13457235 ] Bo Wang commented on HADOOP-8805: - Looked into the testing results of HADOOP-8805-v3.patch. Test failures are not related to changes in this patch. Out of the 4 findbug warnings, 3 of them are in the protocbuf generated code (which we can't change), the rest one is from Configuration class which is unrelated to this patch. Move protocol buffer implementation of GetUserMappingProtocol from HDFS to Common - Key: HADOOP-8805 URL: https://issues.apache.org/jira/browse/HADOOP-8805 Project: Hadoop Common Issue Type: Improvement Reporter: Bo Wang Assignee: Bo Wang Attachments: HADOOP-8805.patch, HADOOP-8805-v2.patch, HADOOP-8805-v3.patch org.apache.hadoop.tools.GetUserMappingProtocol is used in both HDFS and YARN. We should move the protocol buffer implementation from HDFS to Common so that it can also be used by YARN. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
[
https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13457236#comment-13457236
]
Andy Isaacson commented on HADOOP-8806:
---
I didn't intend my exploratory tarball to derail my Friday +1. +1.
libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
Key: HADOOP-8806
URL: https://issues.apache.org/jira/browse/HADOOP-8806
Project: Hadoop Common
Issue Type: Improvement
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Attachments: HADOOP-8806.003.patch, rpathtest2.tar.gz,
rpathtest.tar.gz
libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These
libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For
example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this
directory. However, snappy can't be loaded from this directory unless
{{LD_LIBRARY_PATH}} is set to include this directory.
Can we make this configuration just work without needing to rely on
{{LD_LIBRARY_PATH}}?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8786) HttpServer continues to start even if AuthenticationFilter fails to init
[ https://issues.apache.org/jira/browse/HADOOP-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HADOOP-8786: Attachment: HADOOP-8786-branch-1.patch HttpServer continues to start even if AuthenticationFilter fails to init Key: HADOOP-8786 URL: https://issues.apache.org/jira/browse/HADOOP-8786 Project: Hadoop Common Issue Type: Bug Affects Versions: 1.2.0, 3.0.0, 2.0.1-alpha Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 1.2.0, 3.0.0, 2.0.3-alpha Attachments: HADOOP-8786-branch-1.patch, HADOOP-8786-branch-2.patch, hadoop-8786.txt As seen in HDFS-3904, if the AuthenticationFilter fails to initialize, the web server will continue to start up. We need to check for context initialization errors after starting the server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8763) Set group owner on Windows failed
[
https://issues.apache.org/jira/browse/HADOOP-8763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13457240#comment-13457240
]
Chuan Liu commented on HADOOP-8763:
---
The following code seems to be an unrelated change? Also, do you mean BUILDIN
or BUILTIN?
You are right. It should be BUILTIN. This code is relevant in the sense it
makes the function less prone to potential errors, which is used by 'winutils
chown'.
Do you see any unexpected behavior for users because of the following?
No. I did not see any unexpected behavior. This is just for future references.
We can just leave around the public constant Shell.SET_GROUP_COMMAND or
deprecate it. I am okay leaving it around.
I have left out this in new patch.
Not sure of your usage of asserts vs exit-code, but in src/winutils/chown.c,
instead of asserts for zero-length string, we should log a msg to stderr and
return an EXIT_FAILURE? Also, if both are empty also you should return
EXIT_FAILURE?
The assertions assert for previous parsing code. The parsing code will not
initiate and allocate memory for 'userName' and 'groupName' of zero-length,
i.e. 'userName' and 'groupName' are initiated or NULL in such cases. We should
not return error in such cases because 'chown : file' is a correct usage here,
though no user name or group name is given.
This code won't be invoked on linux, because, ahm, this is winutils? In any
case, that is not behaviour I know, a chown user: filename shouldn't change
the group-name
I have tested the Linux behaviors. You can also check out the man page of
'chown': http://linux.die.net/man/1/chown
Again, as my answer to Bikas's question, this usage pattern is not found in
Hadoop on Linux or Windows that I am aware of. I think it is good to document
the difference here for future reference.
Set group owner on Windows failed
-
Key: HADOOP-8763
URL: https://issues.apache.org/jira/browse/HADOOP-8763
Project: Hadoop Common
Issue Type: Bug
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Minor
Fix For: 1-win
Attachments: HADOOP-8763-branch-1-win-2.patch,
HADOOP-8763-branch-1-win.patch
RawLocalFileSystem.setOwner() method may incorrectly set the group owner of a
file on Windows.
Specifically the following function in RawLocalFileSystem class will fail on
Windows when username is null, i.e. only set group ownership.
{code}
public void setOwner(Path p, String username, String groupname)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8763) Set group owner on Windows failed
[
https://issues.apache.org/jira/browse/HADOOP-8763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chuan Liu updated HADOOP-8763:
--
Attachment: HADOOP-8763-branch-1-win-3.patch
Set group owner on Windows failed
-
Key: HADOOP-8763
URL: https://issues.apache.org/jira/browse/HADOOP-8763
Project: Hadoop Common
Issue Type: Bug
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Minor
Fix For: 1-win
Attachments: HADOOP-8763-branch-1-win-2.patch,
HADOOP-8763-branch-1-win-3.patch, HADOOP-8763-branch-1-win.patch
RawLocalFileSystem.setOwner() method may incorrectly set the group owner of a
file on Windows.
Specifically the following function in RawLocalFileSystem class will fail on
Windows when username is null, i.e. only set group ownership.
{code}
public void setOwner(Path p, String username, String groupname)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8806) libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
[
https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated HADOOP-8806:
Component/s: build
Target Version/s: 2.0.3-alpha
Affects Version/s: 2.0.0-alpha
Hadoop Flags: Reviewed
+1 lgtm as well. Thanks for the deleted explanation guys.
Also, thanks for testing both the java.library.patch and LD_LIBRARY_PATH
scenarios Colin. The test failure and findbugs from the earlier test-patch run
are unrelated to this change. I've tested a native build on my host for sanity.
libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
Key: HADOOP-8806
URL: https://issues.apache.org/jira/browse/HADOOP-8806
Project: Hadoop Common
Issue Type: Improvement
Components: build
Affects Versions: 2.0.0-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Attachments: HADOOP-8806.003.patch, rpathtest2.tar.gz,
rpathtest.tar.gz
libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These
libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For
example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this
directory. However, snappy can't be loaded from this directory unless
{{LD_LIBRARY_PATH}} is set to include this directory.
Can we make this configuration just work without needing to rely on
{{LD_LIBRARY_PATH}}?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8806) libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
[
https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated HADOOP-8806:
Resolution: Fixed
Fix Version/s: 2.0.3-alpha
Target Version/s: (was: 2.0.3-alpha)
Status: Resolved (was: Patch Available)
I've committed this and merged to branch-2. Thanks Colin.
libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
Key: HADOOP-8806
URL: https://issues.apache.org/jira/browse/HADOOP-8806
Project: Hadoop Common
Issue Type: Improvement
Components: build
Affects Versions: 2.0.0-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Fix For: 2.0.3-alpha
Attachments: HADOOP-8806.003.patch, rpathtest2.tar.gz,
rpathtest.tar.gz
libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These
libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For
example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this
directory. However, snappy can't be loaded from this directory unless
{{LD_LIBRARY_PATH}} is set to include this directory.
Can we make this configuration just work without needing to rely on
{{LD_LIBRARY_PATH}}?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8812) ExitUtil#terminate should print Exception#toString
[ https://issues.apache.org/jira/browse/HADOOP-8812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HADOOP-8812: Target Version/s: 2.0.3-alpha (was: 2.0.2-alpha) ExitUtil#terminate should print Exception#toString --- Key: HADOOP-8812 URL: https://issues.apache.org/jira/browse/HADOOP-8812 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.2-alpha Reporter: Eli Collins Assignee: Eli Collins Priority: Minor Attachments: hadoop-8812.txt, hadoop-8812.txt Per Steve's feedback on ExitUtil#terminate should print Exception#toString rather than use getMessage as the latter may return null. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
[
https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13457250#comment-13457250
]
Hudson commented on HADOOP-8806:
Integrated in Hadoop-Common-trunk-Commit #2737 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2737/])
HADOOP-8806. libhadoop.so: dlopen should be better at locating
libsnappy.so, etc. Contributed by Colin Patrick McCabe (Revision 1386784)
HADOOP-8806. libhadoop.so: dlopen should be better at locating libsnappy.so,
etc. Contributed by Colin Patrick McCabe (Revision 1386780)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1386784
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1386780
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ExitUtil.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java
libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
Key: HADOOP-8806
URL: https://issues.apache.org/jira/browse/HADOOP-8806
Project: Hadoop Common
Issue Type: Improvement
Components: build
Affects Versions: 2.0.0-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Fix For: 2.0.3-alpha
Attachments: HADOOP-8806.003.patch, rpathtest2.tar.gz,
rpathtest.tar.gz
libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These
libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For
example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this
directory. However, snappy can't be loaded from this directory unless
{{LD_LIBRARY_PATH}} is set to include this directory.
Can we make this configuration just work without needing to rely on
{{LD_LIBRARY_PATH}}?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8805) Move protocol buffer implementation of GetUserMappingProtocol from HDFS to Common
[ https://issues.apache.org/jira/browse/HADOOP-8805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13457249#comment-13457249 ] Suresh Srinivas commented on HADOOP-8805: - When committing, please ensure that svn mv is used to preserve the history. Move protocol buffer implementation of GetUserMappingProtocol from HDFS to Common - Key: HADOOP-8805 URL: https://issues.apache.org/jira/browse/HADOOP-8805 Project: Hadoop Common Issue Type: Improvement Reporter: Bo Wang Assignee: Bo Wang Attachments: HADOOP-8805.patch, HADOOP-8805-v2.patch, HADOOP-8805-v3.patch org.apache.hadoop.tools.GetUserMappingProtocol is used in both HDFS and YARN. We should move the protocol buffer implementation from HDFS to Common so that it can also be used by YARN. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
[
https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13457251#comment-13457251
]
Hudson commented on HADOOP-8806:
Integrated in Hadoop-Hdfs-trunk-Commit #2800 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2800/])
HADOOP-8806. libhadoop.so: dlopen should be better at locating
libsnappy.so, etc. Contributed by Colin Patrick McCabe (Revision 1386784)
HADOOP-8806. libhadoop.so: dlopen should be better at locating libsnappy.so,
etc. Contributed by Colin Patrick McCabe (Revision 1386780)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1386784
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1386780
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ExitUtil.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java
libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
Key: HADOOP-8806
URL: https://issues.apache.org/jira/browse/HADOOP-8806
Project: Hadoop Common
Issue Type: Improvement
Components: build
Affects Versions: 2.0.0-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Fix For: 2.0.3-alpha
Attachments: HADOOP-8806.003.patch, rpathtest2.tar.gz,
rpathtest.tar.gz
libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These
libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For
example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this
directory. However, snappy can't be loaded from this directory unless
{{LD_LIBRARY_PATH}} is set to include this directory.
Can we make this configuration just work without needing to rely on
{{LD_LIBRARY_PATH}}?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
[
https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13457280#comment-13457280
]
Hudson commented on HADOOP-8806:
Integrated in Hadoop-Mapreduce-trunk-Commit #2761 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2761/])
HADOOP-8806. libhadoop.so: dlopen should be better at locating
libsnappy.so, etc. Contributed by Colin Patrick McCabe (Revision 1386784)
HADOOP-8806. libhadoop.so: dlopen should be better at locating libsnappy.so,
etc. Contributed by Colin Patrick McCabe (Revision 1386780)
Result = FAILURE
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1386784
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1386780
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ExitUtil.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java
libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
Key: HADOOP-8806
URL: https://issues.apache.org/jira/browse/HADOOP-8806
Project: Hadoop Common
Issue Type: Improvement
Components: build
Affects Versions: 2.0.0-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Fix For: 2.0.3-alpha
Attachments: HADOOP-8806.003.patch, rpathtest2.tar.gz,
rpathtest.tar.gz
libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These
libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For
example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this
directory. However, snappy can't be loaded from this directory unless
{{LD_LIBRARY_PATH}} is set to include this directory.
Can we make this configuration just work without needing to rely on
{{LD_LIBRARY_PATH}}?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
[
https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13457370#comment-13457370
]
Roman Shaposhnik commented on HADOOP-8806:
--
Since Linux (GNU really) documentation leaves quite a bit open for
interpretation as far as $ORIGIN and dependencies of unbundled products go,
here's a much more complete treatment of the from Solaris' Linkers and
Libraries doc:
http://docs.oracle.com/cd/E19253-01/817-1984/appendixc-4/index.html
libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
Key: HADOOP-8806
URL: https://issues.apache.org/jira/browse/HADOOP-8806
Project: Hadoop Common
Issue Type: Improvement
Components: build
Affects Versions: 2.0.0-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Fix For: 2.0.3-alpha
Attachments: HADOOP-8806.003.patch, rpathtest2.tar.gz,
rpathtest.tar.gz
libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These
libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For
example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this
directory. However, snappy can't be loaded from this directory unless
{{LD_LIBRARY_PATH}} is set to include this directory.
Can we make this configuration just work without needing to rely on
{{LD_LIBRARY_PATH}}?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8805) Move protocol buffer implementation of GetUserMappingProtocol from HDFS to Common
[ https://issues.apache.org/jira/browse/HADOOP-8805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13457418#comment-13457418 ] Alejandro Abdelnur commented on HADOOP-8805: On the patch: * GetUserMappingProtocol.proto: ** package name 'proto' seems to depart from the naming conventions being use, ti should be 'protobuf' ** class name should remain GetUserMappingsProtocolProtos following current naming conventions. * Why the rename of GetUserMappingsProtocolClientSideTranslatorPB/GetUserMappingsProtocolServerSideTranslatorPB classes? This seems to depart for the current naming convention? * GetUserMappingsProtocolPB: the InterfaceAudience seems it should be only HDFS YARN, not MapReduce, correct? Have you build/deployed and tested it works as expected? Move protocol buffer implementation of GetUserMappingProtocol from HDFS to Common - Key: HADOOP-8805 URL: https://issues.apache.org/jira/browse/HADOOP-8805 Project: Hadoop Common Issue Type: Improvement Reporter: Bo Wang Assignee: Bo Wang Attachments: HADOOP-8805.patch, HADOOP-8805-v2.patch, HADOOP-8805-v3.patch org.apache.hadoop.tools.GetUserMappingProtocol is used in both HDFS and YARN. We should move the protocol buffer implementation from HDFS to Common so that it can also be used by YARN. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8805) Move protocol buffer implementation of GetUserMappingProtocol from HDFS to Common
[ https://issues.apache.org/jira/browse/HADOOP-8805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13457448#comment-13457448 ] Bo Wang commented on HADOOP-8805: - Thanks for the comments, Alejandro. Replies inline. * GetUserMappingProtocol.proto: ** package name 'proto' seems to depart from the naming conventions being use, ti should be 'protobuf' Currently all protobuf generated packages use proto as the name. We can just follow the convention. ** class name should remain GetUserMappingsProtocolProtos following current naming conventions. Sure. * Why the rename of GetUserMappingsProtocolClientSideTranslatorPB/GetUserMappingsProtocolServerSideTranslatorPB classes? This seems to depart for the current naming convention? I was following the YARN naming conventions. We'll keep HDFS naming convention which is along the lines of the naming convention used in Common. * GetUserMappingsProtocolPB: the InterfaceAudience seems it should be only HDFS YARN, not MapReduce, correct? Yes. Will upload a new patch reflecting these feedbacks. Move protocol buffer implementation of GetUserMappingProtocol from HDFS to Common - Key: HADOOP-8805 URL: https://issues.apache.org/jira/browse/HADOOP-8805 Project: Hadoop Common Issue Type: Improvement Reporter: Bo Wang Assignee: Bo Wang Attachments: HADOOP-8805.patch, HADOOP-8805-v2.patch, HADOOP-8805-v3.patch org.apache.hadoop.tools.GetUserMappingProtocol is used in both HDFS and YARN. We should move the protocol buffer implementation from HDFS to Common so that it can also be used by YARN. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-8804) Improve Web UIs when the wildcard address is used
[ https://issues.apache.org/jira/browse/HADOOP-8804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins reassigned HADOOP-8804: --- Assignee: Senthil V Kumar (was: Vinay) Hi Senthil, Not sure we want to use the hostname since presumably that's already in the URL bar. Ie we're getting additional info (the RPC address and port). Some options: - We just have NameNode (active), eg remove the IP:port - We use the NN hostname's IP and RPC port - We use the logical NN id (eg dfs.ha.namenodes.nameserviceId1.nn1) from the config - We leave as is because it's useful to indicate RPC is bound to the wildcard Btw noticed you're patch is using tabs/4 spaces. Please checkout for the style guide: http://wiki.apache.org/hadoop/HowToContributeAlso please don't re-org all the import statements in this change. Thanks, Eli Improve Web UIs when the wildcard address is used - Key: HADOOP-8804 URL: https://issues.apache.org/jira/browse/HADOOP-8804 Project: Hadoop Common Issue Type: Improvement Affects Versions: 1.0.0, 2.0.0-alpha Reporter: Eli Collins Assignee: Senthil V Kumar Priority: Minor Labels: newbie Attachments: HADOOP-8804-1.0.patch, HADOOP-8804-trunk.patch When IPC addresses are bound to the wildcard (ie the default config) the NN, JT (and probably RM etc) Web UIs are a little goofy. Eg 0 Hadoop Map/Reduce Administration and NameNode '0.0.0.0:18021' (active). Let's improve them. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8250) Investigate uses of FileUtil and functional correctness based on current use cases
[ https://issues.apache.org/jira/browse/HADOOP-8250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha resolved HADOOP-8250. Resolution: Won't Fix Multiple other jiras have superceded this. Investigate uses of FileUtil and functional correctness based on current use cases -- Key: HADOOP-8250 URL: https://issues.apache.org/jira/browse/HADOOP-8250 Project: Hadoop Common Issue Type: Sub-task Components: native Affects Versions: 1.1.0 Reporter: Bikas Saha Assignee: Bikas Saha The current Windows patch replaces symlink with copy. This jira tracks understanding the implications of this change and others like it on expected functionality. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8803) Make Hadoop running more secure public cloud envrionment
[ https://issues.apache.org/jira/browse/HADOOP-8803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13457534#comment-13457534 ] Xianqing Yu commented on HADOOP-8803: - Hi Luke, Thank you for your comment. To your first, fourth and fifth comments. My security assumption is that one or some of tasktracker or datanode machines may be compromised by hacker. If HDFS only use one key for all HDFS, even the block tokens are ephemeral and the key only be used for a few minutes, as long as the hacker can get root privilege, she or he always can get the new key from the namenode and generate arbitrary Block Token to access other datanodes, in the end, the whole HDFS is exposed to attacker. The reason datanode have to use unique key is that limiting the hacker capabilities when she or he compromised one datanode. Using TLS is a good way to provide network security. However, you have to assume that all machines are secure or all machine are running under isolated network environment no matter how many clusters are working together. Otherwise, compromised of one machine almost means whole HDFS can be exposed to hacker immediately. For HDFS, once its scale gets bigger and bigger, I feel it is necessary to create certain level of isolation (without sacrifice too much performance of course). Unique key for each datanode is first step for that. But we have to find the best way to lower the overhead. To your second comment, I am working on that test now. Would add more comments when I am done. To your Third comment, thanks for mentioning that, NameNode has to generate token for each replica and that indeed degrade the performance. Another possible way is that let Namenode to choose replica for the client, but NameNode have to perform ChooseDatanode code. I can test how much of this overhead and also check which way has lower overhead. (I am not sure how much overhead it would introduce, will that impact performance once scale of cluster get larger) Make Hadoop running more secure public cloud envrionment Key: HADOOP-8803 URL: https://issues.apache.org/jira/browse/HADOOP-8803 Project: Hadoop Common Issue Type: New Feature Components: fs, ipc, security Affects Versions: 0.20.204.0 Reporter: Xianqing Yu Labels: hadoop Original Estimate: 2m Remaining Estimate: 2m I am a Ph.D student in North Carolina State University. I am modifying the Hadoop's code (which including most parts of Hadoop, e.g. JobTracker, TaskTracker, NameNode, DataNode) to achieve better security. My major goal is that make Hadoop running more secure in the Cloud environment, especially for public Cloud environment. In order to achieve that, I redesign the currently security mechanism and achieve following proprieties: 1. Bring byte-level access control to Hadoop HDFS. Based on 0.20.204, HDFS access control is based on user or block granularity, e.g. HDFS Delegation Token only check if the file can be accessed by certain user or not, Block Token only proof which block or blocks can be accessed. I make Hadoop can do byte-granularity access control, each access party, user or task process can only access the bytes she or he least needed. 2. I assume that in the public Cloud environment, only Namenode, secondary Namenode, JobTracker can be trusted. A large number of Datanode and TaskTracker may be compromised due to some of them may be running under less secure environment. So I re-design the secure mechanism to make the damage the hacker can do to be minimized. a. Re-design the Block Access Token to solve wildly shared-key problem of HDFS. In original Block Access Token design, all HDFS (Namenode and Datanode) share one master key to generate Block Access Token, if one DataNode is compromised by hacker, the hacker can get the key and generate any Block Access Token he or she want. b. Re-design the HDFS Delegation Token to do fine-grain access control for TaskTracker and Map-Reduce Task process on HDFS. In the Hadoop 0.20.204, all TaskTrackers can use their kerberos credentials to access any files for MapReduce on HDFS. So they have the same privilege as JobTracker to do read or write tokens, copy job file, etc.. However, if one of them is compromised, every critical thing in MapReduce directory (job file, Delegation Token) is exposed to attacker. I solve the problem by making JobTracker to decide which TaskTracker can access which file in MapReduce Directory on HDFS. For Task process, once it get HDFS Delegation Token, it can access everything belong to this job or user on HDFS. By my design, it can only access the bytes it needed from HDFS. There are some other improvement in the security,
[jira] [Commented] (HADOOP-8615) EOFException in DecompressorStream.java needs to be more verbose
[
https://issues.apache.org/jira/browse/HADOOP-8615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13457584#comment-13457584
]
thomastechs commented on HADOOP-8615:
-
Thanks Andy for your valuable comments.
I will work on the patch in the trunk too.Also on the test case.
The methods having default values were already existing. I used the same
methods and overloaded with an added parameter of fileName.This will not affect
existing functionalities or the people who already used the existing methods
without a fileName.The new methods with fileName , helps the user to use it, if
the user faces the issue addressed in this bug.
Please let me know your comments.
EOFException in DecompressorStream.java needs to be more verbose
Key: HADOOP-8615
URL: https://issues.apache.org/jira/browse/HADOOP-8615
Project: Hadoop Common
Issue Type: Bug
Components: io
Affects Versions: 0.20.2
Reporter: Jeff Lord
Labels: patch
Attachments: HADOOP-8615-release-0.20.2.patch
In ./src/core/org/apache/hadoop/io/compress/DecompressorStream.java
The following exception should at least pass back the file that it encounters
this error in relation to:
protected void getCompressedData() throws IOException {
checkStream();
int n = in.read(buffer, 0, buffer.length);
if (n == -1) {
throw new EOFException(Unexpected end of input stream);
}
This would help greatly to debug bad/corrupt files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8805) Move protocol buffer implementation of GetUserMappingProtocol from HDFS to Common
[ https://issues.apache.org/jira/browse/HADOOP-8805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13457618#comment-13457618 ] Bo Wang commented on HADOOP-8805: - I realized a problem that YARN RPC requires protocol implementation follows the following naming convention: PROTOCOL + PBServiceImpl. This naming convention is hard coded in org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl#getPbServiceImplClassName. Thus, to make GetUserMappingProtocol usable in both HDFS YARN, we have to follow the convention of YARN rather than HDFS.. In YARN, the protocbuf generated class shares the same name as the protocol itself (e.g. RMAdminProtocol). Thus, the current patch follows YARN convention. I have tested the patch from command line as follows. (1) start a namenode (2) bin/hdfs groups username It returns the correct results. Move protocol buffer implementation of GetUserMappingProtocol from HDFS to Common - Key: HADOOP-8805 URL: https://issues.apache.org/jira/browse/HADOOP-8805 Project: Hadoop Common Issue Type: Improvement Reporter: Bo Wang Assignee: Bo Wang Attachments: HADOOP-8805.patch, HADOOP-8805-v2.patch, HADOOP-8805-v3.patch org.apache.hadoop.tools.GetUserMappingProtocol is used in both HDFS and YARN. We should move the protocol buffer implementation from HDFS to Common so that it can also be used by YARN. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
