[jira] [Commented] (HADOOP-8992) Enhance unit-test coverage of class HarFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-8992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504522#comment-13504522 ] Hudson commented on HADOOP-8992: Integrated in Hadoop-Yarn-trunk #49 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/49/]) HADOOP-8992. Enhance unit-test coverage of class HarFileSystem (Ivan A. Veselovsky via bobby) (Revision 1413743) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1413743 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestHarFileSystemBasics.java Enhance unit-test coverage of class HarFileSystem - Key: HADOOP-8992 URL: https://issues.apache.org/jira/browse/HADOOP-8992 Project: Hadoop Common Issue Type: Improvement Reporter: Ivan A. Veselovsky Assignee: Ivan A. Veselovsky Priority: Minor Fix For: 3.0.0, 2.0.3-alpha, 0.23.6 Attachments: HADOOP-8992-branch-0.23--a.patch, HADOOP-8992-branch-0.23--b.patch, HADOOP-8992-branch-0.23--c.patch, HADOOP-8992-branch-2--a.patch, HADOOP-8992-branch-2--b.patch, HADOOP-8992-branch-2--c.patch New unit test TestHarFileSystem2 provided in order to enhance coverage of class HarFileSystem. Also some unused methods deleted from class HarFileSystem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9093) Move all the Exception in PathExceptions to o.a.h.fs package
[ https://issues.apache.org/jira/browse/HADOOP-9093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504523#comment-13504523 ] Hudson commented on HADOOP-9093: Integrated in Hadoop-Yarn-trunk #49 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/49/]) HADOOP-9093. Move all the Exception in PathExceptions to o.a.h.fs package. Contributed by Suresh Srinivas. (Revision 1413960) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1413960 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathAccessDeniedException.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathExistsException.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathIOException.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathIsDirectoryException.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathIsNotDirectoryException.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathIsNotEmptyDirectoryException.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathNotFoundException.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathOperationException.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathPermissionException.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Command.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CommandWithDestination.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CopyCommands.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Delete.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Display.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Mkdir.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/MoveCommands.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/PathData.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/PathExceptions.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/SetReplication.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Tail.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Test.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Touchz.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestPathExceptions.java Move all the Exception in PathExceptions to o.a.h.fs package Key: HADOOP-9093 URL: https://issues.apache.org/jira/browse/HADOOP-9093 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.2-alpha Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 2.0.3-alpha Attachments: HADOOP-9093.patch The exceptions in PathExceptions are useful for non shell related functionality as well. Making this available as exceptions under fs will help move some of the HDFS implementation code throw more specific exception than throwing IOException (for example see HDFS-4209). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9038) provide unit-test coverage of class org.apache.hadoop.fs.LocalDirAllocator.AllocatorPerContext.PathIterator
[ https://issues.apache.org/jira/browse/HADOOP-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504524#comment-13504524 ] Hudson commented on HADOOP-9038: Integrated in Hadoop-Yarn-trunk #49 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/49/]) HADOOP-9038. unit-tests for AllocatorPerContext.PathIterator (Ivan A. Veselovsky via bobby) (Revision 1413776) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1413776 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalDirAllocator.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalDirAllocator.java provide unit-test coverage of class org.apache.hadoop.fs.LocalDirAllocator.AllocatorPerContext.PathIterator --- Key: HADOOP-9038 URL: https://issues.apache.org/jira/browse/HADOOP-9038 Project: Hadoop Common Issue Type: Test Reporter: Ivan A. Veselovsky Assignee: Ivan A. Veselovsky Priority: Minor Fix For: 3.0.0, 2.0.3-alpha, 0.23.6 Attachments: HADOOP-9038.patch The class org.apache.hadoop.fs.LocalDirAllocator.AllocatorPerContext.PathIterator currently has zero unit-test coverage. Add/enhance the tests to provide one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9097) Maven RAT plugin is not checking all source files
Tom White created HADOOP-9097: - Summary: Maven RAT plugin is not checking all source files Key: HADOOP-9097 URL: https://issues.apache.org/jira/browse/HADOOP-9097 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 2.0.3-alpha, 0.23.5 Reporter: Tom White Priority: Blocker Fix For: 2.0.3-alpha, 0.23.6 Running 'mvn apache-rat:check' passes, but running RAT by hand (by downloading the JAR) produces some warnings for Java files, amongst others. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9097) Maven RAT plugin is not checking all source files
[
https://issues.apache.org/jira/browse/HADOOP-9097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504572#comment-13504572
]
Tom White commented on HADOOP-9097:
---
It looks like the plugin is configured to only check pom.xml in some places:
{noformat}
plugin
groupIdorg.apache.rat/groupId
artifactIdapache-rat-plugin/artifactId
configuration
includes
includepom.xml/include
/includes
/configuration
/plugin
{noformat}
We should change this to include everything by default, and list exclusions in
cases where it is not possible to add a license header (e.g. binary files).
Service loader files (under META-INF), can have headers added since # is
recognized as a comment.
Maven RAT plugin is not checking all source files
-
Key: HADOOP-9097
URL: https://issues.apache.org/jira/browse/HADOOP-9097
Project: Hadoop Common
Issue Type: Bug
Components: build
Affects Versions: 2.0.3-alpha, 0.23.5
Reporter: Tom White
Priority: Blocker
Fix For: 2.0.3-alpha, 0.23.6
Running 'mvn apache-rat:check' passes, but running RAT by hand (by
downloading the JAR) produces some warnings for Java files, amongst others.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9016) Provide unit tests for class org.apache.hadoop.fs.HarFileSystem.HarFSDataInputStream
[ https://issues.apache.org/jira/browse/HADOOP-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan A. Veselovsky updated HADOOP-9016: --- Attachment: HADOOP-9016--d.patch HADOOP-9016-branch-0.23--d.patch The attached patches HADOOP-9016-branch-0.23--d.patch, HADOOP-9016--d.patch are fixed to apply correctly to the current HEAD repository state. The patch HADOOP-9016--d.patch targeted to branches trunk and branch-2. The patch HADOOP-9016-branch-0.23--d.patch targeted to branch branch-0.23. Provide unit tests for class org.apache.hadoop.fs.HarFileSystem.HarFSDataInputStream - Key: HADOOP-9016 URL: https://issues.apache.org/jira/browse/HADOOP-9016 Project: Hadoop Common Issue Type: Improvement Reporter: Ivan A. Veselovsky Assignee: Ivan A. Veselovsky Priority: Minor Attachments: HADOOP-9016--b.patch, HADOOP-9016-branch-0.23--d.patch, HADOOP-9016--c.patch, HADOOP-9016--d.patch, HADOOP-9016.patch unit-test coverage of classes org.apache.hadoop.fs.HarFileSystem.HarFSDataInputStream, org.apache.hadoop.fs.HarFileSystem.HarFSDataInputStream.HarFsInputStream is zero. Suggested to provide unit-tests covering these classes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9038) provide unit-test coverage of class org.apache.hadoop.fs.LocalDirAllocator.AllocatorPerContext.PathIterator
[ https://issues.apache.org/jira/browse/HADOOP-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504583#comment-13504583 ] Hudson commented on HADOOP-9038: Integrated in Hadoop-Hdfs-0.23-Build #448 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/448/]) svn merge -c 1413776 FIXES: HADOOP-9038. unit-tests for AllocatorPerContext.PathIterator (Ivan A. Veselovsky via bobby) (Revision 1413779) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1413779 Files : * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalDirAllocator.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalDirAllocator.java provide unit-test coverage of class org.apache.hadoop.fs.LocalDirAllocator.AllocatorPerContext.PathIterator --- Key: HADOOP-9038 URL: https://issues.apache.org/jira/browse/HADOOP-9038 Project: Hadoop Common Issue Type: Test Reporter: Ivan A. Veselovsky Assignee: Ivan A. Veselovsky Priority: Minor Fix For: 3.0.0, 2.0.3-alpha, 0.23.6 Attachments: HADOOP-9038.patch The class org.apache.hadoop.fs.LocalDirAllocator.AllocatorPerContext.PathIterator currently has zero unit-test coverage. Add/enhance the tests to provide one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8992) Enhance unit-test coverage of class HarFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-8992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504582#comment-13504582 ] Hudson commented on HADOOP-8992: Integrated in Hadoop-Hdfs-0.23-Build #448 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/448/]) svn merge -c 1413743 FIXES: HADOOP-8992. Enhance unit-test coverage of class HarFileSystem (Ivan A. Veselovsky via bobby) (Revision 1413746) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1413746 Files : * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestHarFileSystemBasics.java Enhance unit-test coverage of class HarFileSystem - Key: HADOOP-8992 URL: https://issues.apache.org/jira/browse/HADOOP-8992 Project: Hadoop Common Issue Type: Improvement Reporter: Ivan A. Veselovsky Assignee: Ivan A. Veselovsky Priority: Minor Fix For: 3.0.0, 2.0.3-alpha, 0.23.6 Attachments: HADOOP-8992-branch-0.23--a.patch, HADOOP-8992-branch-0.23--b.patch, HADOOP-8992-branch-0.23--c.patch, HADOOP-8992-branch-2--a.patch, HADOOP-8992-branch-2--b.patch, HADOOP-8992-branch-2--c.patch New unit test TestHarFileSystem2 provided in order to enhance coverage of class HarFileSystem. Also some unused methods deleted from class HarFileSystem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9098) Add missing license headers
Tom White created HADOOP-9098: - Summary: Add missing license headers Key: HADOOP-9098 URL: https://issues.apache.org/jira/browse/HADOOP-9098 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 1.1.1 Reporter: Tom White Priority: Blocker Fix For: 1.2.0 There are missing license headers in some source files (e.g. TestUnderReplicatedBlocks.java is one) according to the RAT report. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9093) Move all the Exception in PathExceptions to o.a.h.fs package
[ https://issues.apache.org/jira/browse/HADOOP-9093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504591#comment-13504591 ] Hudson commented on HADOOP-9093: Integrated in Hadoop-Hdfs-trunk #1239 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1239/]) HADOOP-9093. Move all the Exception in PathExceptions to o.a.h.fs package. Contributed by Suresh Srinivas. (Revision 1413960) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1413960 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathAccessDeniedException.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathExistsException.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathIOException.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathIsDirectoryException.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathIsNotDirectoryException.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathIsNotEmptyDirectoryException.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathNotFoundException.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathOperationException.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathPermissionException.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Command.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CommandWithDestination.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CopyCommands.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Delete.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Display.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Mkdir.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/MoveCommands.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/PathData.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/PathExceptions.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/SetReplication.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Tail.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Test.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Touchz.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestPathExceptions.java Move all the Exception in PathExceptions to o.a.h.fs package Key: HADOOP-9093 URL: https://issues.apache.org/jira/browse/HADOOP-9093 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.2-alpha Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 2.0.3-alpha Attachments: HADOOP-9093.patch The exceptions in PathExceptions are useful for non shell related functionality as well. Making this available as exceptions under fs will help move some of the HDFS implementation code throw more specific exception than throwing IOException (for example see HDFS-4209). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9038) provide unit-test coverage of class org.apache.hadoop.fs.LocalDirAllocator.AllocatorPerContext.PathIterator
[ https://issues.apache.org/jira/browse/HADOOP-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504592#comment-13504592 ] Hudson commented on HADOOP-9038: Integrated in Hadoop-Hdfs-trunk #1239 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1239/]) HADOOP-9038. unit-tests for AllocatorPerContext.PathIterator (Ivan A. Veselovsky via bobby) (Revision 1413776) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1413776 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalDirAllocator.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalDirAllocator.java provide unit-test coverage of class org.apache.hadoop.fs.LocalDirAllocator.AllocatorPerContext.PathIterator --- Key: HADOOP-9038 URL: https://issues.apache.org/jira/browse/HADOOP-9038 Project: Hadoop Common Issue Type: Test Reporter: Ivan A. Veselovsky Assignee: Ivan A. Veselovsky Priority: Minor Fix For: 3.0.0, 2.0.3-alpha, 0.23.6 Attachments: HADOOP-9038.patch The class org.apache.hadoop.fs.LocalDirAllocator.AllocatorPerContext.PathIterator currently has zero unit-test coverage. Add/enhance the tests to provide one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9016) Provide unit tests for class org.apache.hadoop.fs.HarFileSystem.HarFSDataInputStream
[
https://issues.apache.org/jira/browse/HADOOP-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504595#comment-13504595
]
Hadoop QA commented on HADOOP-9016:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12555003/HADOOP-9016--d.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-tools/hadoop-archives.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1827//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1827//console
This message is automatically generated.
Provide unit tests for class
org.apache.hadoop.fs.HarFileSystem.HarFSDataInputStream
-
Key: HADOOP-9016
URL: https://issues.apache.org/jira/browse/HADOOP-9016
Project: Hadoop Common
Issue Type: Improvement
Reporter: Ivan A. Veselovsky
Assignee: Ivan A. Veselovsky
Priority: Minor
Attachments: HADOOP-9016--b.patch, HADOOP-9016-branch-0.23--d.patch,
HADOOP-9016--c.patch, HADOOP-9016--d.patch, HADOOP-9016.patch
unit-test coverage of classes
org.apache.hadoop.fs.HarFileSystem.HarFSDataInputStream,
org.apache.hadoop.fs.HarFileSystem.HarFSDataInputStream.HarFsInputStream is
zero.
Suggested to provide unit-tests covering these classes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8992) Enhance unit-test coverage of class HarFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-8992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504607#comment-13504607 ] Hudson commented on HADOOP-8992: Integrated in Hadoop-Mapreduce-trunk #1270 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1270/]) HADOOP-8992. Enhance unit-test coverage of class HarFileSystem (Ivan A. Veselovsky via bobby) (Revision 1413743) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1413743 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestHarFileSystemBasics.java Enhance unit-test coverage of class HarFileSystem - Key: HADOOP-8992 URL: https://issues.apache.org/jira/browse/HADOOP-8992 Project: Hadoop Common Issue Type: Improvement Reporter: Ivan A. Veselovsky Assignee: Ivan A. Veselovsky Priority: Minor Fix For: 3.0.0, 2.0.3-alpha, 0.23.6 Attachments: HADOOP-8992-branch-0.23--a.patch, HADOOP-8992-branch-0.23--b.patch, HADOOP-8992-branch-0.23--c.patch, HADOOP-8992-branch-2--a.patch, HADOOP-8992-branch-2--b.patch, HADOOP-8992-branch-2--c.patch New unit test TestHarFileSystem2 provided in order to enhance coverage of class HarFileSystem. Also some unused methods deleted from class HarFileSystem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9093) Move all the Exception in PathExceptions to o.a.h.fs package
[ https://issues.apache.org/jira/browse/HADOOP-9093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504608#comment-13504608 ] Hudson commented on HADOOP-9093: Integrated in Hadoop-Mapreduce-trunk #1270 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1270/]) HADOOP-9093. Move all the Exception in PathExceptions to o.a.h.fs package. Contributed by Suresh Srinivas. (Revision 1413960) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1413960 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathAccessDeniedException.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathExistsException.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathIOException.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathIsDirectoryException.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathIsNotDirectoryException.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathIsNotEmptyDirectoryException.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathNotFoundException.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathOperationException.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathPermissionException.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Command.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CommandWithDestination.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CopyCommands.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Delete.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Display.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Mkdir.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/MoveCommands.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/PathData.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/PathExceptions.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/SetReplication.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Tail.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Test.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Touchz.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestPathExceptions.java Move all the Exception in PathExceptions to o.a.h.fs package Key: HADOOP-9093 URL: https://issues.apache.org/jira/browse/HADOOP-9093 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.2-alpha Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 2.0.3-alpha Attachments: HADOOP-9093.patch The exceptions in PathExceptions are useful for non shell related functionality as well. Making this available as exceptions under fs will help move some of the HDFS implementation code throw more specific exception than throwing IOException (for example see HDFS-4209). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9038) provide unit-test coverage of class org.apache.hadoop.fs.LocalDirAllocator.AllocatorPerContext.PathIterator
[ https://issues.apache.org/jira/browse/HADOOP-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504609#comment-13504609 ] Hudson commented on HADOOP-9038: Integrated in Hadoop-Mapreduce-trunk #1270 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1270/]) HADOOP-9038. unit-tests for AllocatorPerContext.PathIterator (Ivan A. Veselovsky via bobby) (Revision 1413776) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1413776 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalDirAllocator.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalDirAllocator.java provide unit-test coverage of class org.apache.hadoop.fs.LocalDirAllocator.AllocatorPerContext.PathIterator --- Key: HADOOP-9038 URL: https://issues.apache.org/jira/browse/HADOOP-9038 Project: Hadoop Common Issue Type: Test Reporter: Ivan A. Veselovsky Assignee: Ivan A. Veselovsky Priority: Minor Fix For: 3.0.0, 2.0.3-alpha, 0.23.6 Attachments: HADOOP-9038.patch The class org.apache.hadoop.fs.LocalDirAllocator.AllocatorPerContext.PathIterator currently has zero unit-test coverage. Add/enhance the tests to provide one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9046) provide unit-test coverage of class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT
[ https://issues.apache.org/jira/browse/HADOOP-9046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504626#comment-13504626 ] Ivan A. Veselovsky commented on HADOOP-9046: Hi, Robert, in last pre-commit verification build (above) the test org.apache.hadoop.fs.TestDelegationTokenRenewer took 39 seconds: https://builds.apache.org/job/PreCommit-HADOOP-Build/1821//testReport/org.apache.hadoop.fs/TestDelegationTokenRenewer/ (both the test cases). in my desktop the console mvn run of this test takes 38 sec (i do just mvn clean test -Dtest=org.apache.hadoop.fs.TestDelegationTokenRenewer). When I run it from within Eclipse, it also takes 18 + 21 = 39 sec. Looking to the code I didn't realize how this test could take 7 minutes. So, can you please provide more details on the conditions of your experiment: 1) did you use the attached HADOOP-9046-over-9049.patch against trunk? 2) what JDK you ran the test upon? 3) did the both test cases pass? 4) do other Hadoop tests on your machine run with speed similar to that in Apache Jenkins build machine (https://builds.apache.org) ? 5) what exact command line did you use to run the test(s)? (e.g. was -Pnative included?) 6) is that effect (7 min) reproducible on that machine? 7) There are 2 test cases in the class org.apache.hadoop.fs.TestDelegationTokenRenewer. Which one was slow? Or both? Thanks in advance. provide unit-test coverage of class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT -- Key: HADOOP-9046 URL: https://issues.apache.org/jira/browse/HADOOP-9046 Project: Hadoop Common Issue Type: Test Reporter: Ivan A. Veselovsky Assignee: Ivan A. Veselovsky Priority: Minor Attachments: HADOOP-9046-branch-0.23-over-9049.patch, HADOOP-9046-branch-0.23.patch, HADOOP-9046-over-9049.patch, HADOOP-9046.patch The class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT has zero coverage in entire cumulative test run. Provide test(s) to cover this class. Note: the request submitted to HDFS project because the class likely to be tested by tests in that project. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9046) provide unit-test coverage of class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT
[ https://issues.apache.org/jira/browse/HADOOP-9046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504627#comment-13504627 ] Ivan A. Veselovsky commented on HADOOP-9046: One more note: the code that waits for the FS weak ref to dispose executed with 30 sec timeout, so it looks like it cannot be the cause of 7 min slowdown. provide unit-test coverage of class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT -- Key: HADOOP-9046 URL: https://issues.apache.org/jira/browse/HADOOP-9046 Project: Hadoop Common Issue Type: Test Reporter: Ivan A. Veselovsky Assignee: Ivan A. Veselovsky Priority: Minor Attachments: HADOOP-9046-branch-0.23-over-9049.patch, HADOOP-9046-branch-0.23.patch, HADOOP-9046-over-9049.patch, HADOOP-9046.patch The class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT has zero coverage in entire cumulative test run. Provide test(s) to cover this class. Note: the request submitted to HDFS project because the class likely to be tested by tests in that project. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9093) Move all the Exception in PathExceptions to o.a.h.fs package
[
https://issues.apache.org/jira/browse/HADOOP-9093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504648#comment-13504648
]
Daryn Sharp commented on HADOOP-9093:
-
bq. I propose using FileNotFoundException instead of PathNotFoundException as
it is already extensively used. Similarly use AccessControlException instead of
PathAccessException. If folks agree, I will make that change in the next patch.
Alternatively we could at least make these exceptions subclasses of the
exception that I am proposing replacing them with.
I had considered that when I created these exceptions, but wanted all path
exceptions to derive from a common class. I suppose {{PathException}} could be
an interface and we copy-n-paste the base code - which is the main factor I
chose to derive from a base class.
Move all the Exception in PathExceptions to o.a.h.fs package
Key: HADOOP-9093
URL: https://issues.apache.org/jira/browse/HADOOP-9093
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 2.0.2-alpha
Reporter: Suresh Srinivas
Assignee: Suresh Srinivas
Fix For: 2.0.3-alpha
Attachments: HADOOP-9093.patch
The exceptions in PathExceptions are useful for non shell related
functionality as well. Making this available as exceptions under fs will help
move some of the HDFS implementation code throw more specific exception than
throwing IOException (for example see HDFS-4209).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9046) provide unit-test coverage of class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT
[ https://issues.apache.org/jira/browse/HADOOP-9046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504691#comment-13504691 ] Ivan A. Veselovsky commented on HADOOP-9046: Hi, Robert, looks like I found the reason of delay: the main thread locks for long time when waiting to enter the synchronized method org.apache.hadoop.fs.DelegationTokenRenewer.removeRenewAction(T), and this operation is invoked in StateSynchronizer *without* a timeout. I will update the patch when i fix this. Thanks a lot for this catch. provide unit-test coverage of class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT -- Key: HADOOP-9046 URL: https://issues.apache.org/jira/browse/HADOOP-9046 Project: Hadoop Common Issue Type: Test Reporter: Ivan A. Veselovsky Assignee: Ivan A. Veselovsky Priority: Minor Attachments: HADOOP-9046-branch-0.23-over-9049.patch, HADOOP-9046-branch-0.23.patch, HADOOP-9046-over-9049.patch, HADOOP-9046.patch The class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT has zero coverage in entire cumulative test run. Provide test(s) to cover this class. Note: the request submitted to HDFS project because the class likely to be tested by tests in that project. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9092) Coverage fixing for org.apache.hadoop.mapreduce.jobhistory
[ https://issues.apache.org/jira/browse/HADOOP-9092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504694#comment-13504694 ] Robert Joseph Evans commented on HADOOP-9092: - The change looks good. I have one minor comment still. in TestJobHistoryEventHandler there is a line where you are using ...+ to convert something to a string. This works, but I would prefer to see something more like Stirng.valueOf(...) instead. Other then that I am +1 for the patch. Coverage fixing for org.apache.hadoop.mapreduce.jobhistory --- Key: HADOOP-9092 URL: https://issues.apache.org/jira/browse/HADOOP-9092 Project: Hadoop Common Issue Type: Test Components: tools Reporter: Aleksey Gorshkov Attachments: HADOOP-9092-branch-0.23.patch, HADOOP-9092-branch-2.patch, HADOOP-9092-trunk.patch Coverage fixing for package org.apache.hadoop.mapreduce.jobhistory -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9083) Port HADOOP-9020 Add a SASL PLAIN server to branch 1
[ https://issues.apache.org/jira/browse/HADOOP-9083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504701#comment-13504701 ] Daryn Sharp commented on HADOOP-9083: - Is this intended to be the first of a series of jiras to port PLAIN auth into 1.x? I've not completed the implementation on trunk/branch-2, so this jira doesn't provide all the support necessary to actually use PLAIN auth via RPC. Based on a quick read of the the hive jira, it sounds as if perhaps hive only intends to directly use the PLAIN server? Port HADOOP-9020 Add a SASL PLAIN server to branch 1 Key: HADOOP-9083 URL: https://issues.apache.org/jira/browse/HADOOP-9083 Project: Hadoop Common Issue Type: Task Components: ipc, security Affects Versions: 1.0.3 Reporter: Yu Gao Assignee: Yu Gao Attachments: HADOOP-9020-branch-1.patch, test-patch.result, test-TestSaslRPC.result It would be good if the patch of HADOOP-9020 for adding SASL PLAIN server implementation could be ported to branch 1 as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9046) provide unit-test coverage of class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT
[ https://issues.apache.org/jira/browse/HADOOP-9046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504760#comment-13504760 ] Robert Joseph Evans commented on HADOOP-9046: - I am glad that you caught it. Sorry I did not respond sooner. provide unit-test coverage of class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT -- Key: HADOOP-9046 URL: https://issues.apache.org/jira/browse/HADOOP-9046 Project: Hadoop Common Issue Type: Test Reporter: Ivan A. Veselovsky Assignee: Ivan A. Veselovsky Priority: Minor Attachments: HADOOP-9046-branch-0.23-over-9049.patch, HADOOP-9046-branch-0.23.patch, HADOOP-9046-over-9049.patch, HADOOP-9046.patch The class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT has zero coverage in entire cumulative test run. Provide test(s) to cover this class. Note: the request submitted to HDFS project because the class likely to be tested by tests in that project. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9090) Refactor MetricsSystemImpl to allow for an on-demand publish system
[ https://issues.apache.org/jira/browse/HADOOP-9090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504783#comment-13504783 ] Luke Lu commented on HADOOP-9090: - I can imagine that you would want to control publishing the metrics at the end of a process. What about a longer running process that also want to periodically publish its metrics? Wouldn't it be better to enhance the metrics system to expose an ondemand publish and wait (until the metrics are sent or timed out) method, so the same metrics system can work with all processes? long running or not? Refactor MetricsSystemImpl to allow for an on-demand publish system --- Key: HADOOP-9090 URL: https://issues.apache.org/jira/browse/HADOOP-9090 Project: Hadoop Common Issue Type: New Feature Components: metrics Reporter: Mostafa Elhemali Priority: Minor Attachments: HADOOP-9090.2.patch, HADOOP-9090.patch We have a need to publish metrics out of some short-living processes, which is not really well-suited to the current metrics system implementation which periodically publishes metrics asynchronously (a behavior that works great for long-living processes). Of course I could write my own metrics system, but it seems like such a waste to rewrite all the awesome code currently in the MetricsSystemImpl and supporting classes. The way I'm proposing to solve this is to: 1. Refactor the MetricsSystemImpl class into an abstract base MetricsSystemImpl class (common configuration and other code) and a concrete PeriodicPublishMetricsSystemImpl class (timer thread). 2. Refactor the MetricsSinkAdapter class into an abstract base MetricsSinkAdapter class (common configuration and other code) and a concrete AsyncMetricsSinkAdapter class (asynchronous publishing using the SinkQueue). 3. Derive a new simple class OnDemandPublishMetricsSystemImpl from MetricsSystemImpl, that just exposes a synchronous publish() method to do all the work. 4. Derive a SyncMetricsSinkAdapter class from MetricsSinkAdapter to just synchronously push metrics to the underlying sink. Does that sound reasonable? I'll attach the patch with all this coded up and simple tests (could use some polish I guess, but wanted to get everyone's opinion first). Notice that this is somewhat of a breaking change since MetricsSystemImpl is public (although it's marked with InterfaceAudience.Private); if the breaking change is a problem I could just rename the refactored classes so that PeriodicPublishMetricsSystemImpl is still called MetricsSystemImpl (and MetricsSystemImpl - BaseMetricsSystemImpl). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8981) TestMetricsSystemImpl fails on Windows
[
https://issues.apache.org/jira/browse/HADOOP-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504810#comment-13504810
]
Xuan Gong commented on HADOOP-8981:
---
For this test failure, I do not think our code has any problems. So, basically,
how the testInitFirst function works like this: it will register three sinks,
named Test,Sink1 and Sink2, each of them has its own blockingQueue,
initially the queue is empty, and if there are any items in the queue, they
will consume them. In the testInitFirst function, what the function
ms.onTimerEvent() did is to put items to all of the queues that let the sinks
can consume the items. Followed by this function is stop function. Basically it
stops all the threads.
Why the test is fail is because the sink does not finish to consume the items,
it is stopped by calling the function stop(). If we add Thread.sleep(sometime)
before the stop function, the test will be successful.Original I think maybe we
can add the checking condition to check if the queue is empty before we
actually do the stop. But I think that the purpose of this stop() function is
when I call this functions, all the sink threads should stop no matter they are
idle or they are doing something. If we actually add the checking condition or
other synchronized mechnism to make sure all the items in queue is consumed
before we stop the thread, does this violate the purpose of calling stop
function ?
And actually the test is fail, on the other side, it proved that our stop
function is actually worked correctly. When we call the stop, everything is
stopped no matter what they are doing right now. Based on that, I runned this
test several times, I found that sometimes it will be successful, sometimes
half rigth half wrong, sometimes all fails.
If we try to make the testcase pass, what we need to do is to change the test
case to :
verify(sink1, atMost(2)).putMetrics(r1.capture());
ListMetricsRecord mr1 = r1.getAllValues();
verify(sink2, atMost(2)).putMetrics(r2.capture());
ListMetricsRecord mr2 = r2.getAllValues();
if(mr1.size() != 0 mr2.size() != 0){
checkMetricsRecords(mr1);
assertEquals(output, mr1, mr2);
}else if(mr1.size() != 0){
checkMetricsRecords(mr1);
}else if(mr2.size() != 0){
checkMetricsRecords(mr2);
}
TestMetricsSystemImpl fails on Windows
--
Key: HADOOP-8981
URL: https://issues.apache.org/jira/browse/HADOOP-8981
Project: Hadoop Common
Issue Type: Bug
Components: metrics
Affects Versions: trunk-win
Reporter: Chris Nauroth
Assignee: Arpit Agarwal
The test is failing on an expected mock interaction.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9088) Add Murmur3 hash
[ https://issues.apache.org/jira/browse/HADOOP-9088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504824#comment-13504824 ] Todd Lipcon commented on HADOOP-9088: - Hi Radim. We don't currently use the murmur2 hash anywhere AFAIK. Where do you anticipate wanting to use this new hash function? Add Murmur3 hash Key: HADOOP-9088 URL: https://issues.apache.org/jira/browse/HADOOP-9088 Project: Hadoop Common Issue Type: Improvement Reporter: Radim Kolar Assignee: Radim Kolar Attachments: murmur3-2.txt, murmur3-3.txt, murmur3.txt faster and better then murmur2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9064) Augment DelegationTokenRenewer API to cancel the tokens on calls to removeRenewAction
[
https://issues.apache.org/jira/browse/HADOOP-9064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504840#comment-13504840
]
Alejandro Abdelnur commented on HADOOP-9064:
+1
Augment DelegationTokenRenewer API to cancel the tokens on calls to
removeRenewAction
-
Key: HADOOP-9064
URL: https://issues.apache.org/jira/browse/HADOOP-9064
Project: Hadoop Common
Issue Type: Bug
Components: security
Affects Versions: 2.0.2-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Attachments: hadoop-9064.patch, hadoop-9064.patch, hadoop-9064.patch,
hadoop-9064.patch
Post HADOOP-9049, FileSystems register with DelegationTokenRenewer (a
singleton), to renew tokens.
To avoid a bunch of defunct tokens clog the NN, we should augment the API to
{{#removeRenewAction(boolean cancel)}} and cancel the token appropriately.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9064) Augment DelegationTokenRenewer API to cancel the tokens on calls to removeRenewAction
[
https://issues.apache.org/jira/browse/HADOOP-9064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated HADOOP-9064:
---
Resolution: Fixed
Fix Version/s: 2.0.3-alpha
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks Karthik. Committed to trunk and branch-2.
Augment DelegationTokenRenewer API to cancel the tokens on calls to
removeRenewAction
-
Key: HADOOP-9064
URL: https://issues.apache.org/jira/browse/HADOOP-9064
Project: Hadoop Common
Issue Type: Bug
Components: security
Affects Versions: 2.0.2-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Fix For: 2.0.3-alpha
Attachments: hadoop-9064.patch, hadoop-9064.patch, hadoop-9064.patch,
hadoop-9064.patch
Post HADOOP-9049, FileSystems register with DelegationTokenRenewer (a
singleton), to renew tokens.
To avoid a bunch of defunct tokens clog the NN, we should augment the API to
{{#removeRenewAction(boolean cancel)}} and cancel the token appropriately.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9090) Refactor MetricsSystemImpl to allow for an on-demand publish system
[
https://issues.apache.org/jira/browse/HADOOP-9090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504862#comment-13504862
]
Hadoop QA commented on HADOOP-9090:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12555046/HADOOP-9090.justEnhanceDefaultImpl.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1828//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1828//console
This message is automatically generated.
Refactor MetricsSystemImpl to allow for an on-demand publish system
---
Key: HADOOP-9090
URL: https://issues.apache.org/jira/browse/HADOOP-9090
Project: Hadoop Common
Issue Type: New Feature
Components: metrics
Reporter: Mostafa Elhemali
Priority: Minor
Attachments: HADOOP-9090.2.patch,
HADOOP-9090.justEnhanceDefaultImpl.patch, HADOOP-9090.patch
We have a need to publish metrics out of some short-living processes, which
is not really well-suited to the current metrics system implementation which
periodically publishes metrics asynchronously (a behavior that works great
for long-living processes). Of course I could write my own metrics system,
but it seems like such a waste to rewrite all the awesome code currently in
the MetricsSystemImpl and supporting classes.
The way I'm proposing to solve this is to:
1. Refactor the MetricsSystemImpl class into an abstract base
MetricsSystemImpl class (common configuration and other code) and a concrete
PeriodicPublishMetricsSystemImpl class (timer thread).
2. Refactor the MetricsSinkAdapter class into an abstract base
MetricsSinkAdapter class (common configuration and other code) and a concrete
AsyncMetricsSinkAdapter class (asynchronous publishing using the SinkQueue).
3. Derive a new simple class OnDemandPublishMetricsSystemImpl from
MetricsSystemImpl, that just exposes a synchronous publish() method to do all
the work.
4. Derive a SyncMetricsSinkAdapter class from MetricsSinkAdapter to just
synchronously push metrics to the underlying sink.
Does that sound reasonable? I'll attach the patch with all this coded up and
simple tests (could use some polish I guess, but wanted to get everyone's
opinion first). Notice that this is somewhat of a breaking change since
MetricsSystemImpl is public (although it's marked with
InterfaceAudience.Private); if the breaking change is a problem I could just
rename the refactored classes so that PeriodicPublishMetricsSystemImpl is
still called MetricsSystemImpl (and MetricsSystemImpl -
BaseMetricsSystemImpl).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9088) Add Murmur3 hash
[ https://issues.apache.org/jira/browse/HADOOP-9088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504865#comment-13504865 ] Radim Kolar commented on HADOOP-9088: - to improve hashing. WritableComparator hash is weak (used by binarypartitioner), HashPartitioner sucks even more (using object.hashcode), i will add TextPartitioner using murmur3, it have good distribution of results. Add Murmur3 hash Key: HADOOP-9088 URL: https://issues.apache.org/jira/browse/HADOOP-9088 Project: Hadoop Common Issue Type: Improvement Reporter: Radim Kolar Assignee: Radim Kolar Attachments: murmur3-2.txt, murmur3-3.txt, murmur3.txt faster and better then murmur2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9084) TotalOrderPartitioner fails on hadoop running on top of gpfs (or any parallel or distributed filesystem)
[ https://issues.apache.org/jira/browse/HADOOP-9084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504868#comment-13504868 ] Hudson commented on HADOOP-9084: Integrated in Hadoop-trunk-Commit #3062 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3062/]) HADOOP-9084. Augment DelegationTokenRenewer API to cancel the tokens on calls to removeRenewAction. (kkambatl via tucu) (Revision 1414326) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1414326 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegationTokenRenewer.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestDelegationTokenRenewer.java TotalOrderPartitioner fails on hadoop running on top of gpfs (or any parallel or distributed filesystem) Key: HADOOP-9084 URL: https://issues.apache.org/jira/browse/HADOOP-9084 Project: Hadoop Common Issue Type: Bug Components: filecache, fs, native Affects Versions: 1.0.3, 1.0.4 Reporter: giovanni delussu Assignee: giovanni delussu Priority: Critical Fix For: 1.0.3, 1.0.4 Attachments: PATCH-HADOOP-9084-origin-branch-1.0.patch When running a job who uses TotalOrderPartitioner (like TeraSort or BulkImport of HBase) on hadoop running on top of gpfs (instead of hdfs) the program fails to find the file _partition.lst because is looking for it in the wrong directory. The confusion is between local fs meaning not hdfs and local meaning distributed fs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9056) Build native library on Windows
[ https://issues.apache.org/jira/browse/HADOOP-9056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504872#comment-13504872 ] Chuan Liu commented on HADOOP-9056: --- Looks good in general! I have the following questions and comments. # It seems this patch only contains changes that made compiling ‘hadoop.dll’ available on Windows. Some functionality was missing. For example, ‘POSIX.chmod’ was not ported over from branch-1-win; so were some other IO methods. I also notice some new functions were introduced in trunk compared with branch-1. For example, posixFadviseIfPossible() and POSIX.posix_fadvis(). What is the plan for those missing functions? Do we plan to port them over in other JIRAs? # “native.vcxproj.user” file is not needed. # Snappy does not work on Windows right now. It may be better to exclude snappy related files from Windows build. # I think it is better to use ‘WINDOWS’ instead of ‘ _WIN32’ macro in some places because we explicitly defined the ‘UNIX’ and ‘WINDOWS’ macros in the beginning of “org_apache_hadoop.h”. It will make it easier in the future if we want to change the definition of the macros. # Some changes in SecureIOUtils and Datanode are not ported over. Again I think this is related to point 1 above. Build native library on Windows --- Key: HADOOP-9056 URL: https://issues.apache.org/jira/browse/HADOOP-9056 Project: Hadoop Common Issue Type: Improvement Components: native Affects Versions: trunk-win Reporter: Arpit Agarwal Assignee: Arpit Agarwal Fix For: trunk-win Attachments: HADOOP-9056.patch Original Estimate: 168h Remaining Estimate: 168h The native library (hadoop.dll) must be compiled on Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9084) TotalOrderPartitioner fails on hadoop running on top of gpfs (or any parallel or distributed filesystem)
[ https://issues.apache.org/jira/browse/HADOOP-9084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504882#comment-13504882 ] Hudson commented on HADOOP-9084: Integrated in Hadoop-trunk-Commit #3063 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3063/]) HADOOP-9064. Ameding CHANGES.txt with correct JIRA ID, wrongfully used HADOOP-9084 before (tucu) (Revision 1414347) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1414347 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt TotalOrderPartitioner fails on hadoop running on top of gpfs (or any parallel or distributed filesystem) Key: HADOOP-9084 URL: https://issues.apache.org/jira/browse/HADOOP-9084 Project: Hadoop Common Issue Type: Bug Components: filecache, fs, native Affects Versions: 1.0.3, 1.0.4 Reporter: giovanni delussu Assignee: giovanni delussu Priority: Critical Fix For: 1.0.3, 1.0.4 Attachments: PATCH-HADOOP-9084-origin-branch-1.0.patch When running a job who uses TotalOrderPartitioner (like TeraSort or BulkImport of HBase) on hadoop running on top of gpfs (instead of hdfs) the program fails to find the file _partition.lst because is looking for it in the wrong directory. The confusion is between local fs meaning not hdfs and local meaning distributed fs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9064) Augment DelegationTokenRenewer API to cancel the tokens on calls to removeRenewAction
[
https://issues.apache.org/jira/browse/HADOOP-9064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504883#comment-13504883
]
Hudson commented on HADOOP-9064:
Integrated in Hadoop-trunk-Commit #3063 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/3063/])
HADOOP-9064. Ameding CHANGES.txt with correct JIRA ID, wrongfully used
HADOOP-9084 before (tucu) (Revision 1414347)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1414347
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
Augment DelegationTokenRenewer API to cancel the tokens on calls to
removeRenewAction
-
Key: HADOOP-9064
URL: https://issues.apache.org/jira/browse/HADOOP-9064
Project: Hadoop Common
Issue Type: Bug
Components: security
Affects Versions: 2.0.2-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Fix For: 2.0.3-alpha
Attachments: hadoop-9064.patch, hadoop-9064.patch, hadoop-9064.patch,
hadoop-9064.patch
Post HADOOP-9049, FileSystems register with DelegationTokenRenewer (a
singleton), to renew tokens.
To avoid a bunch of defunct tokens clog the NN, we should augment the API to
{{#removeRenewAction(boolean cancel)}} and cancel the token appropriately.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9088) Add Murmur3 hash
[ https://issues.apache.org/jira/browse/HADOOP-9088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504886#comment-13504886 ] Todd Lipcon commented on HADOOP-9088: - I don't think HashPartitioner can be changed to use anything but Object.hashCode (which we assume the user has implemented). Changing WritableComparator to use a different hash hardly seems worth it. Have you seen cases where the existing hash code causes poor distribution? Also, I don't think we can change the default hashcode for existing types, since it would change user-visible partitioning behavior. Adding a new TextPartitioner which uses Murmur3 sounds useful, if you can show that there are indeed real-world datasets where the poor hash behavior of the existing partitioner causes skew. Add Murmur3 hash Key: HADOOP-9088 URL: https://issues.apache.org/jira/browse/HADOOP-9088 Project: Hadoop Common Issue Type: Improvement Reporter: Radim Kolar Assignee: Radim Kolar Attachments: murmur3-2.txt, murmur3-3.txt, murmur3.txt faster and better then murmur2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9093) Move all the Exception in PathExceptions to o.a.h.fs package
[ https://issues.apache.org/jira/browse/HADOOP-9093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504891#comment-13504891 ] Tsz Wo (Nicholas), SZE commented on HADOOP-9093: ... I suppose PathException could be an interface ... I also have thought about changing the base class to interface. However, it may not be very useful since we cannot catch interface as it no longer extends Throwable. Move all the Exception in PathExceptions to o.a.h.fs package Key: HADOOP-9093 URL: https://issues.apache.org/jira/browse/HADOOP-9093 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.2-alpha Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 2.0.3-alpha Attachments: HADOOP-9093.patch The exceptions in PathExceptions are useful for non shell related functionality as well. Making this available as exceptions under fs will help move some of the HDFS implementation code throw more specific exception than throwing IOException (for example see HDFS-4209). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7868) Hadoop native fails to compile when default linker option is -Wl,--as-needed
[ https://issues.apache.org/jira/browse/HADOOP-7868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated HADOOP-7868: -- Fix Version/s: 0.23.6 I pulled this into branch-0.23 Hadoop native fails to compile when default linker option is -Wl,--as-needed Key: HADOOP-7868 URL: https://issues.apache.org/jira/browse/HADOOP-7868 Project: Hadoop Common Issue Type: Bug Components: native Affects Versions: 1.0.0, 2.0.0-alpha Environment: Ubuntu Precise, Ubuntu Oneiric, Debian Unstable Reporter: James Page Assignee: Trevor Robinson Fix For: 1.2.0, 2.0.2-alpha, 0.23.6 Attachments: hadoop-7868-b1.txt, HADOOP-7868.patch, HADOOP-7868-portable.patch Recent releases of Ubuntu and Debian have switched to using --as-needed as default when linking binaries. As a result the AC_COMPUTE_NEEDED_DSO fails to find the required DSO names during execution of configure resulting in a build failure. Explicitly using -Wl,--no-as-needed in this macro when required resolves this issue. See http://wiki.debian.org/ToolChain/DSOLinking for a few more details -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9088) Add Murmur3 hash
[ https://issues.apache.org/jira/browse/HADOOP-9088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504919#comment-13504919 ] Radim Kolar commented on HADOOP-9088: - Hash partitioner can do same thing like java hashtable - rehashing hashCode() to get better distribution. current hash quality in hadoop is low if you have lot of similar strings like aab you will get about 20% unoptimal partitions in average cases. but in some specific cases it can split like 80:20 istead of close 50:50 Add Murmur3 hash Key: HADOOP-9088 URL: https://issues.apache.org/jira/browse/HADOOP-9088 Project: Hadoop Common Issue Type: Improvement Reporter: Radim Kolar Assignee: Radim Kolar Attachments: murmur3-2.txt, murmur3-3.txt, murmur3.txt faster and better then murmur2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9090) Refactor MetricsSystemImpl to allow for an on-demand publish system
[ https://issues.apache.org/jira/browse/HADOOP-9090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504926#comment-13504926 ] Luke Lu commented on HADOOP-9090: - The new patch is indeed much simpler :) I don't think the sinkLock is needed though, as publishMetrics is already synchronized. Another thing is error handling with publishMetricsNow. What if underlying sink hangs (due to network issues)? Do you want the process to potentially hang forever if the final metrics are not published? Refactor MetricsSystemImpl to allow for an on-demand publish system --- Key: HADOOP-9090 URL: https://issues.apache.org/jira/browse/HADOOP-9090 Project: Hadoop Common Issue Type: New Feature Components: metrics Reporter: Mostafa Elhemali Priority: Minor Attachments: HADOOP-9090.2.patch, HADOOP-9090.justEnhanceDefaultImpl.patch, HADOOP-9090.patch We have a need to publish metrics out of some short-living processes, which is not really well-suited to the current metrics system implementation which periodically publishes metrics asynchronously (a behavior that works great for long-living processes). Of course I could write my own metrics system, but it seems like such a waste to rewrite all the awesome code currently in the MetricsSystemImpl and supporting classes. The way I'm proposing to solve this is to: 1. Refactor the MetricsSystemImpl class into an abstract base MetricsSystemImpl class (common configuration and other code) and a concrete PeriodicPublishMetricsSystemImpl class (timer thread). 2. Refactor the MetricsSinkAdapter class into an abstract base MetricsSinkAdapter class (common configuration and other code) and a concrete AsyncMetricsSinkAdapter class (asynchronous publishing using the SinkQueue). 3. Derive a new simple class OnDemandPublishMetricsSystemImpl from MetricsSystemImpl, that just exposes a synchronous publish() method to do all the work. 4. Derive a SyncMetricsSinkAdapter class from MetricsSinkAdapter to just synchronously push metrics to the underlying sink. Does that sound reasonable? I'll attach the patch with all this coded up and simple tests (could use some polish I guess, but wanted to get everyone's opinion first). Notice that this is somewhat of a breaking change since MetricsSystemImpl is public (although it's marked with InterfaceAudience.Private); if the breaking change is a problem I could just rename the refactored classes so that PeriodicPublishMetricsSystemImpl is still called MetricsSystemImpl (and MetricsSystemImpl - BaseMetricsSystemImpl). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9088) Add Murmur3 hash
[ https://issues.apache.org/jira/browse/HADOOP-9088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar updated HADOOP-9088: Attachment: murmur3-4.txt faster variant from mahout Add Murmur3 hash Key: HADOOP-9088 URL: https://issues.apache.org/jira/browse/HADOOP-9088 Project: Hadoop Common Issue Type: Improvement Reporter: Radim Kolar Assignee: Radim Kolar Attachments: murmur3-2.txt, murmur3-3.txt, murmur3-4.txt, murmur3.txt faster and better then murmur2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9088) Add Murmur3 hash
[ https://issues.apache.org/jira/browse/HADOOP-9088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504941#comment-13504941 ] Radim Kolar commented on HADOOP-9088: - bloom filter code is using util.Hash family of functions Add Murmur3 hash Key: HADOOP-9088 URL: https://issues.apache.org/jira/browse/HADOOP-9088 Project: Hadoop Common Issue Type: Improvement Reporter: Radim Kolar Assignee: Radim Kolar Attachments: murmur3-2.txt, murmur3-3.txt, murmur3-4.txt, murmur3.txt faster and better then murmur2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9088) Add Murmur3 hash
[
https://issues.apache.org/jira/browse/HADOOP-9088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504945#comment-13504945
]
Hadoop QA commented on HADOOP-9088:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12555062/murmur3-4.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1829//console
This message is automatically generated.
Add Murmur3 hash
Key: HADOOP-9088
URL: https://issues.apache.org/jira/browse/HADOOP-9088
Project: Hadoop Common
Issue Type: Improvement
Reporter: Radim Kolar
Assignee: Radim Kolar
Attachments: murmur3-2.txt, murmur3-3.txt, murmur3-4.txt, murmur3.txt
faster and better then murmur2
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9088) Add Murmur3 hash
[ https://issues.apache.org/jira/browse/HADOOP-9088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar updated HADOOP-9088: Attachment: murmur3-5.txt Add Murmur3 hash Key: HADOOP-9088 URL: https://issues.apache.org/jira/browse/HADOOP-9088 Project: Hadoop Common Issue Type: Improvement Reporter: Radim Kolar Assignee: Radim Kolar Attachments: murmur3-2.txt, murmur3-3.txt, murmur3-4.txt, murmur3-5.txt, murmur3.txt faster and better then murmur2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9088) Add Murmur3 hash
[
https://issues.apache.org/jira/browse/HADOOP-9088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504955#comment-13504955
]
Hadoop QA commented on HADOOP-9088:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12555064/murmur3-5.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1830//console
This message is automatically generated.
Add Murmur3 hash
Key: HADOOP-9088
URL: https://issues.apache.org/jira/browse/HADOOP-9088
Project: Hadoop Common
Issue Type: Improvement
Reporter: Radim Kolar
Assignee: Radim Kolar
Attachments: murmur3-2.txt, murmur3-3.txt, murmur3-4.txt,
murmur3-5.txt, murmur3.txt
faster and better then murmur2
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9088) Add Murmur3 hash
[ https://issues.apache.org/jira/browse/HADOOP-9088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar updated HADOOP-9088: Attachment: murmur3-6.txt Add Murmur3 hash Key: HADOOP-9088 URL: https://issues.apache.org/jira/browse/HADOOP-9088 Project: Hadoop Common Issue Type: Improvement Reporter: Radim Kolar Assignee: Radim Kolar Attachments: murmur3-2.txt, murmur3-3.txt, murmur3-4.txt, murmur3-5.txt, murmur3-6.txt, murmur3.txt faster and better then murmur2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9090) Refactor MetricsSystemImpl to allow for an on-demand publish system
[ https://issues.apache.org/jira/browse/HADOOP-9090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mostafa Elhemali updated HADOOP-9090: - Attachment: HADOOP-9090.justEnhanceDefaultImpl.2.patch Thanks Luke. I've attached a new patch that guards against sinks taking too long (10 seconds by default) when consuming any single record in the publishMetricsNow() call. I do need to synchronize the publishing of metrics because publishNow() competes with the queue consumption thread for the sink. I've removed sinkLock though and just made the inner method synchronized (on the sink adapter itself) since no one else synchronizes on it so we there should be no contention. Refactor MetricsSystemImpl to allow for an on-demand publish system --- Key: HADOOP-9090 URL: https://issues.apache.org/jira/browse/HADOOP-9090 Project: Hadoop Common Issue Type: New Feature Components: metrics Reporter: Mostafa Elhemali Priority: Minor Attachments: HADOOP-9090.2.patch, HADOOP-9090.justEnhanceDefaultImpl.2.patch, HADOOP-9090.justEnhanceDefaultImpl.patch, HADOOP-9090.patch We have a need to publish metrics out of some short-living processes, which is not really well-suited to the current metrics system implementation which periodically publishes metrics asynchronously (a behavior that works great for long-living processes). Of course I could write my own metrics system, but it seems like such a waste to rewrite all the awesome code currently in the MetricsSystemImpl and supporting classes. The way I'm proposing to solve this is to: 1. Refactor the MetricsSystemImpl class into an abstract base MetricsSystemImpl class (common configuration and other code) and a concrete PeriodicPublishMetricsSystemImpl class (timer thread). 2. Refactor the MetricsSinkAdapter class into an abstract base MetricsSinkAdapter class (common configuration and other code) and a concrete AsyncMetricsSinkAdapter class (asynchronous publishing using the SinkQueue). 3. Derive a new simple class OnDemandPublishMetricsSystemImpl from MetricsSystemImpl, that just exposes a synchronous publish() method to do all the work. 4. Derive a SyncMetricsSinkAdapter class from MetricsSinkAdapter to just synchronously push metrics to the underlying sink. Does that sound reasonable? I'll attach the patch with all this coded up and simple tests (could use some polish I guess, but wanted to get everyone's opinion first). Notice that this is somewhat of a breaking change since MetricsSystemImpl is public (although it's marked with InterfaceAudience.Private); if the breaking change is a problem I could just rename the refactored classes so that PeriodicPublishMetricsSystemImpl is still called MetricsSystemImpl (and MetricsSystemImpl - BaseMetricsSystemImpl). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9088) Add Murmur3 hash
[
https://issues.apache.org/jira/browse/HADOOP-9088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505008#comment-13505008
]
Hadoop QA commented on HADOOP-9088:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12555068/murmur3-6.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1831//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1831//console
This message is automatically generated.
Add Murmur3 hash
Key: HADOOP-9088
URL: https://issues.apache.org/jira/browse/HADOOP-9088
Project: Hadoop Common
Issue Type: Improvement
Reporter: Radim Kolar
Assignee: Radim Kolar
Attachments: murmur3-2.txt, murmur3-3.txt, murmur3-4.txt,
murmur3-5.txt, murmur3-6.txt, murmur3.txt
faster and better then murmur2
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9090) Refactor MetricsSystemImpl to allow for an on-demand publish system
[
https://issues.apache.org/jira/browse/HADOOP-9090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505015#comment-13505015
]
Hadoop QA commented on HADOOP-9090:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12555070/HADOOP-9090.justEnhanceDefaultImpl.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1832//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1832//console
This message is automatically generated.
Refactor MetricsSystemImpl to allow for an on-demand publish system
---
Key: HADOOP-9090
URL: https://issues.apache.org/jira/browse/HADOOP-9090
Project: Hadoop Common
Issue Type: New Feature
Components: metrics
Reporter: Mostafa Elhemali
Priority: Minor
Attachments: HADOOP-9090.2.patch,
HADOOP-9090.justEnhanceDefaultImpl.2.patch,
HADOOP-9090.justEnhanceDefaultImpl.patch, HADOOP-9090.patch
We have a need to publish metrics out of some short-living processes, which
is not really well-suited to the current metrics system implementation which
periodically publishes metrics asynchronously (a behavior that works great
for long-living processes). Of course I could write my own metrics system,
but it seems like such a waste to rewrite all the awesome code currently in
the MetricsSystemImpl and supporting classes.
The way I'm proposing to solve this is to:
1. Refactor the MetricsSystemImpl class into an abstract base
MetricsSystemImpl class (common configuration and other code) and a concrete
PeriodicPublishMetricsSystemImpl class (timer thread).
2. Refactor the MetricsSinkAdapter class into an abstract base
MetricsSinkAdapter class (common configuration and other code) and a concrete
AsyncMetricsSinkAdapter class (asynchronous publishing using the SinkQueue).
3. Derive a new simple class OnDemandPublishMetricsSystemImpl from
MetricsSystemImpl, that just exposes a synchronous publish() method to do all
the work.
4. Derive a SyncMetricsSinkAdapter class from MetricsSinkAdapter to just
synchronously push metrics to the underlying sink.
Does that sound reasonable? I'll attach the patch with all this coded up and
simple tests (could use some polish I guess, but wanted to get everyone's
opinion first). Notice that this is somewhat of a breaking change since
MetricsSystemImpl is public (although it's marked with
InterfaceAudience.Private); if the breaking change is a problem I could just
rename the refactored classes so that PeriodicPublishMetricsSystemImpl is
still called MetricsSystemImpl (and MetricsSystemImpl -
BaseMetricsSystemImpl).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9099) NetUtils.normalizeHostName fails on domains where UnknownHost resolves to an IP address
Ivan Mitic created HADOOP-9099: -- Summary: NetUtils.normalizeHostName fails on domains where UnknownHost resolves to an IP address Key: HADOOP-9099 URL: https://issues.apache.org/jira/browse/HADOOP-9099 Project: Hadoop Common Issue Type: Bug Affects Versions: 1-win Reporter: Ivan Mitic Assignee: Ivan Mitic I just hit this failure. We should use some more unique string for UnknownHost: Testcase: testNormalizeHostName took 0.007 sec FAILED expected:[65.53.5.181] but was:[UnknownHost] junit.framework.AssertionFailedError: expected:[65.53.5.181] but was:[UnknownHost] at org.apache.hadoop.net.TestNetUtils.testNormalizeHostName(TestNetUtils.java:347) Will post a patch in a bit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9099) NetUtils.normalizeHostName fails on domains where UnknownHost resolves to an IP address
[ https://issues.apache.org/jira/browse/HADOOP-9099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Mitic updated HADOOP-9099: --- Attachment: HADOOP-9099.branch-1-win.patch Attaching the patch. NetUtils.normalizeHostName fails on domains where UnknownHost resolves to an IP address --- Key: HADOOP-9099 URL: https://issues.apache.org/jira/browse/HADOOP-9099 Project: Hadoop Common Issue Type: Bug Affects Versions: 1-win Reporter: Ivan Mitic Assignee: Ivan Mitic Attachments: HADOOP-9099.branch-1-win.patch I just hit this failure. We should use some more unique string for UnknownHost: Testcase: testNormalizeHostName took 0.007 sec FAILED expected:[65.53.5.181] but was:[UnknownHost] junit.framework.AssertionFailedError: expected:[65.53.5.181] but was:[UnknownHost] at org.apache.hadoop.net.TestNetUtils.testNormalizeHostName(TestNetUtils.java:347) Will post a patch in a bit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9099) NetUtils.normalizeHostName fails on domains where UnknownHost resolves to an IP address
[ https://issues.apache.org/jira/browse/HADOOP-9099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505062#comment-13505062 ] Mostafa Elhemali commented on HADOOP-9099: -- Personally I would've used something even more unlikely, such as ThisIsNotARealHostName-ItJustCant, but I guess UnknownHost123 is not very likely to exist :). +1 (non-binding). NetUtils.normalizeHostName fails on domains where UnknownHost resolves to an IP address --- Key: HADOOP-9099 URL: https://issues.apache.org/jira/browse/HADOOP-9099 Project: Hadoop Common Issue Type: Bug Affects Versions: 1-win Reporter: Ivan Mitic Assignee: Ivan Mitic Attachments: HADOOP-9099.branch-1-win.patch I just hit this failure. We should use some more unique string for UnknownHost: Testcase: testNormalizeHostName took 0.007 sec FAILED expected:[65.53.5.181] but was:[UnknownHost] junit.framework.AssertionFailedError: expected:[65.53.5.181] but was:[UnknownHost] at org.apache.hadoop.net.TestNetUtils.testNormalizeHostName(TestNetUtils.java:347) Will post a patch in a bit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8989) hadoop dfs -find feature
[ https://issues.apache.org/jira/browse/HADOOP-8989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505074#comment-13505074 ] wolfgang hoschek commented on HADOOP-8989: -- I gave it a try and this patch is awesome. Would be really useful to add the following options as defined here: http://linux.die.net/man/1/find -regextype java (no need for other regex flavours here, but should be possible to add classic flavours later) -regex -maxdepth -mindepth -printf hadoop dfs -find feature Key: HADOOP-8989 URL: https://issues.apache.org/jira/browse/HADOOP-8989 Project: Hadoop Common Issue Type: New Feature Reporter: Marco Nicosia Assignee: Jonathan Allen Attachments: HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch Both sysadmins and users make frequent use of the unix 'find' command, but Hadoop has no correlate. Without this, users are writing scripts which make heavy use of hadoop dfs -lsr, and implementing find one-offs. I think hdfs -lsr is somewhat taxing on the NameNode, and a really slow experience on the client side. Possibly an in-NameNode find operation would be only a bit more taxing on the NameNode, but significantly faster from the client's point of view? The minimum set of options I can think of which would make a Hadoop find command generally useful is (in priority order): * -type (file or directory, for now) * -atime/-ctime-mtime (... and -creationtime?) (both + and - arguments) * -print0 (for piping to xargs -0) * -depth * -owner/-group (and -nouser/-nogroup) * -name (allowing for shell pattern, or even regex?) * -perm * -size One possible special case, but could possibly be really cool if it ran from within the NameNode: * -delete The hadoop dfs -lsr | hadoop dfs -rm cycle is really, really slow. Lower priority, some people do use operators, mostly to execute -or searches such as: * find / \(-nouser -or -nogroup\) Finally, I thought I'd include a link to the [Posix spec for find|http://www.opengroup.org/onlinepubs/009695399/utilities/find.html] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9099) NetUtils.normalizeHostName fails on domains where UnknownHost resolves to an IP address
[ https://issues.apache.org/jira/browse/HADOOP-9099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505079#comment-13505079 ] Tsz Wo (Nicholas), SZE commented on HADOOP-9099: TestNetUtils in branch-1-win does not fail in my machine. Is there any special setting in your machine/network? BTW, I just have a quick check of the InetAddress source code (http://www.docjar.com/html/api/java/net/InetAddress.java.html). It seems that it won't do any lookup if the input string is already an numerical IP address. However, it is probably jdk dependent. NetUtils.normalizeHostName fails on domains where UnknownHost resolves to an IP address --- Key: HADOOP-9099 URL: https://issues.apache.org/jira/browse/HADOOP-9099 Project: Hadoop Common Issue Type: Bug Affects Versions: 1-win Reporter: Ivan Mitic Assignee: Ivan Mitic Attachments: HADOOP-9099.branch-1-win.patch I just hit this failure. We should use some more unique string for UnknownHost: Testcase: testNormalizeHostName took 0.007 sec FAILED expected:[65.53.5.181] but was:[UnknownHost] junit.framework.AssertionFailedError: expected:[65.53.5.181] but was:[UnknownHost] at org.apache.hadoop.net.TestNetUtils.testNormalizeHostName(TestNetUtils.java:347) Will post a patch in a bit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8989) hadoop dfs -find feature
[ https://issues.apache.org/jira/browse/HADOOP-8989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505080#comment-13505080 ] Allen Wittenauer commented on HADOOP-8989: -- Has anyone studied the impact on the NN yet? hadoop dfs -find feature Key: HADOOP-8989 URL: https://issues.apache.org/jira/browse/HADOOP-8989 Project: Hadoop Common Issue Type: New Feature Reporter: Marco Nicosia Assignee: Jonathan Allen Attachments: HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch Both sysadmins and users make frequent use of the unix 'find' command, but Hadoop has no correlate. Without this, users are writing scripts which make heavy use of hadoop dfs -lsr, and implementing find one-offs. I think hdfs -lsr is somewhat taxing on the NameNode, and a really slow experience on the client side. Possibly an in-NameNode find operation would be only a bit more taxing on the NameNode, but significantly faster from the client's point of view? The minimum set of options I can think of which would make a Hadoop find command generally useful is (in priority order): * -type (file or directory, for now) * -atime/-ctime-mtime (... and -creationtime?) (both + and - arguments) * -print0 (for piping to xargs -0) * -depth * -owner/-group (and -nouser/-nogroup) * -name (allowing for shell pattern, or even regex?) * -perm * -size One possible special case, but could possibly be really cool if it ran from within the NameNode: * -delete The hadoop dfs -lsr | hadoop dfs -rm cycle is really, really slow. Lower priority, some people do use operators, mostly to execute -or searches such as: * find / \(-nouser -or -nogroup\) Finally, I thought I'd include a link to the [Posix spec for find|http://www.opengroup.org/onlinepubs/009695399/utilities/find.html] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9099) NetUtils.normalizeHostName fails on domains where UnknownHost resolves to an IP address
[ https://issues.apache.org/jira/browse/HADOOP-9099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505082#comment-13505082 ] Tsz Wo (Nicholas), SZE commented on HADOOP-9099: I understand the problem after looked at Ivan's patch - UnknownHost was resolved to an IP in your setting. NetUtils.normalizeHostName fails on domains where UnknownHost resolves to an IP address --- Key: HADOOP-9099 URL: https://issues.apache.org/jira/browse/HADOOP-9099 Project: Hadoop Common Issue Type: Bug Affects Versions: 1-win Reporter: Ivan Mitic Assignee: Ivan Mitic Attachments: HADOOP-9099.branch-1-win.patch I just hit this failure. We should use some more unique string for UnknownHost: Testcase: testNormalizeHostName took 0.007 sec FAILED expected:[65.53.5.181] but was:[UnknownHost] junit.framework.AssertionFailedError: expected:[65.53.5.181] but was:[UnknownHost] at org.apache.hadoop.net.TestNetUtils.testNormalizeHostName(TestNetUtils.java:347) Will post a patch in a bit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9099) NetUtils.normalizeHostName fails on domains where UnknownHost resolves to an IP address
[ https://issues.apache.org/jira/browse/HADOOP-9099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HADOOP-9099: --- Component/s: test Priority: Minor (was: Major) Hadoop Flags: Reviewed +1 patch looks good. NetUtils.normalizeHostName fails on domains where UnknownHost resolves to an IP address --- Key: HADOOP-9099 URL: https://issues.apache.org/jira/browse/HADOOP-9099 Project: Hadoop Common Issue Type: Bug Components: test Affects Versions: 1-win Reporter: Ivan Mitic Assignee: Ivan Mitic Priority: Minor Attachments: HADOOP-9099.branch-1-win.patch I just hit this failure. We should use some more unique string for UnknownHost: Testcase: testNormalizeHostName took 0.007 sec FAILED expected:[65.53.5.181] but was:[UnknownHost] junit.framework.AssertionFailedError: expected:[65.53.5.181] but was:[UnknownHost] at org.apache.hadoop.net.TestNetUtils.testNormalizeHostName(TestNetUtils.java:347) Will post a patch in a bit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9099) NetUtils.normalizeHostName fails on domains where UnknownHost resolves to an IP address
[ https://issues.apache.org/jira/browse/HADOOP-9099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE resolved HADOOP-9099. Resolution: Fixed Fix Version/s: 1-win 1.2.0 I have committed this. Thanks, Ivan! NetUtils.normalizeHostName fails on domains where UnknownHost resolves to an IP address --- Key: HADOOP-9099 URL: https://issues.apache.org/jira/browse/HADOOP-9099 Project: Hadoop Common Issue Type: Bug Components: test Affects Versions: 1-win Reporter: Ivan Mitic Assignee: Ivan Mitic Priority: Minor Fix For: 1.2.0, 1-win Attachments: HADOOP-9099.branch-1-win.patch I just hit this failure. We should use some more unique string for UnknownHost: Testcase: testNormalizeHostName took 0.007 sec FAILED expected:[65.53.5.181] but was:[UnknownHost] junit.framework.AssertionFailedError: expected:[65.53.5.181] but was:[UnknownHost] at org.apache.hadoop.net.TestNetUtils.testNormalizeHostName(TestNetUtils.java:347) Will post a patch in a bit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9100) Fix TODO items in FsShell
Suresh Srinivas created HADOOP-9100: --- Summary: Fix TODO items in FsShell Key: HADOOP-9100 URL: https://issues.apache.org/jira/browse/HADOOP-9100 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Reporter: Suresh Srinivas Assignee: Suresh Srinivas Priority: Minor Bunch of TODO items in FsShell needs to be fixed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8989) hadoop dfs -find feature
[ https://issues.apache.org/jira/browse/HADOOP-8989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505130#comment-13505130 ] wolfgang hoschek commented on HADOOP-8989: -- In addition, would be good to have a -starttime dateTime dateTimePattern option to express the time related options (e.g. -mmin, -mtime, -amin, -atime) to work relative to a specific absolute timestamp (say, give me all files modified since yesterday midnight) instead of relative to whatever now happens to be at the time of command execution. The dateTimePattern might be ISO 8601 by default, and could be set to any java.text.SimpleDateFormat format pattern. The FindOptions class already seems to foresee such usage. It's just a matter of exposing it at the command level. hadoop dfs -find feature Key: HADOOP-8989 URL: https://issues.apache.org/jira/browse/HADOOP-8989 Project: Hadoop Common Issue Type: New Feature Reporter: Marco Nicosia Assignee: Jonathan Allen Attachments: HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch Both sysadmins and users make frequent use of the unix 'find' command, but Hadoop has no correlate. Without this, users are writing scripts which make heavy use of hadoop dfs -lsr, and implementing find one-offs. I think hdfs -lsr is somewhat taxing on the NameNode, and a really slow experience on the client side. Possibly an in-NameNode find operation would be only a bit more taxing on the NameNode, but significantly faster from the client's point of view? The minimum set of options I can think of which would make a Hadoop find command generally useful is (in priority order): * -type (file or directory, for now) * -atime/-ctime-mtime (... and -creationtime?) (both + and - arguments) * -print0 (for piping to xargs -0) * -depth * -owner/-group (and -nouser/-nogroup) * -name (allowing for shell pattern, or even regex?) * -perm * -size One possible special case, but could possibly be really cool if it ran from within the NameNode: * -delete The hadoop dfs -lsr | hadoop dfs -rm cycle is really, really slow. Lower priority, some people do use operators, mostly to execute -or searches such as: * find / \(-nouser -or -nogroup\) Finally, I thought I'd include a link to the [Posix spec for find|http://www.opengroup.org/onlinepubs/009695399/utilities/find.html] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9018) Reject invalid Windows URIs
[ https://issues.apache.org/jira/browse/HADOOP-9018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-9018: -- Attachment: HADOOP-9018.patch Patch to address some of the issues raised in reference to HADOOP-8953 and HADOOP-8977. Makes path parsing stricter on Windows. Reject invalid Windows URIs --- Key: HADOOP-9018 URL: https://issues.apache.org/jira/browse/HADOOP-9018 Project: Hadoop Common Issue Type: Improvement Components: fs Affects Versions: trunk-win Environment: Windows Reporter: Arpit Agarwal Assignee: Arpit Agarwal Attachments: HADOOP-9018.patch Original Estimate: 24h Remaining Estimate: 24h This JIRA is to make handling of improperly constructed file URIs for Windows local paths more rigorous. e.g. reject file:///c:\\Windows Valid file URI syntax explained at http://blogs.msdn.com/b/ie/archive/2006/12/06/file-uris-in-windows.aspx. Also see https://issues.apache.org/jira/browse/HADOOP-8953 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8532) [Configuration] Increase or make variable substitution depth configurable
[ https://issues.apache.org/jira/browse/HADOOP-8532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505245#comment-13505245 ] Aaron T. Myers commented on HADOOP-8532: Raising it to a higher value is fine by me. [Configuration] Increase or make variable substitution depth configurable - Key: HADOOP-8532 URL: https://issues.apache.org/jira/browse/HADOOP-8532 Project: Hadoop Common Issue Type: Improvement Components: conf Affects Versions: 2.0.0-alpha Reporter: Harsh J We've had some users recently complain that the default MAX_SUBST hardcode of 20 isn't sufficient for their substitution needs and they wished it were configurable rather than having to roll about with workarounds such as using temporary smaller substitutes and then building the fuller one after it. We should consider raising the default hardcode, or provide a way to make it configurable instead. Related: HIVE-2021 changed something similar for their HiveConf classes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9092) Coverage fixing for org.apache.hadoop.mapreduce.jobhistory
[ https://issues.apache.org/jira/browse/HADOOP-9092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Gorshkov updated HADOOP-9092: - Attachment: HADOOP-9092-trunk-a.patch HADOOP-9092-branch-2-a.patch HADOOP-9092-branch-0.23-a.patch Coverage fixing for org.apache.hadoop.mapreduce.jobhistory --- Key: HADOOP-9092 URL: https://issues.apache.org/jira/browse/HADOOP-9092 Project: Hadoop Common Issue Type: Test Components: tools Reporter: Aleksey Gorshkov Attachments: HADOOP-9092-branch-0.23-a.patch, HADOOP-9092-branch-0.23.patch, HADOOP-9092-branch-2-a.patch, HADOOP-9092-branch-2.patch, HADOOP-9092-trunk-a.patch, HADOOP-9092-trunk.patch Coverage fixing for package org.apache.hadoop.mapreduce.jobhistory -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9092) Coverage fixing for org.apache.hadoop.mapreduce.jobhistory
[ https://issues.apache.org/jira/browse/HADOOP-9092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505262#comment-13505262 ] Aleksey Gorshkov commented on HADOOP-9092: -- I've corrected code Coverage fixing for org.apache.hadoop.mapreduce.jobhistory --- Key: HADOOP-9092 URL: https://issues.apache.org/jira/browse/HADOOP-9092 Project: Hadoop Common Issue Type: Test Components: tools Reporter: Aleksey Gorshkov Attachments: HADOOP-9092-branch-0.23-a.patch, HADOOP-9092-branch-0.23.patch, HADOOP-9092-branch-2-a.patch, HADOOP-9092-branch-2.patch, HADOOP-9092-trunk-a.patch, HADOOP-9092-trunk.patch Coverage fixing for package org.apache.hadoop.mapreduce.jobhistory -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9092) Coverage fixing for org.apache.hadoop.mapreduce.jobhistory
[
https://issues.apache.org/jira/browse/HADOOP-9092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505274#comment-13505274
]
Hadoop QA commented on HADOOP-9092:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12555123/HADOOP-9092-trunk-a.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1833//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1833//console
This message is automatically generated.
Coverage fixing for org.apache.hadoop.mapreduce.jobhistory
---
Key: HADOOP-9092
URL: https://issues.apache.org/jira/browse/HADOOP-9092
Project: Hadoop Common
Issue Type: Test
Components: tools
Reporter: Aleksey Gorshkov
Attachments: HADOOP-9092-branch-0.23-a.patch,
HADOOP-9092-branch-0.23.patch, HADOOP-9092-branch-2-a.patch,
HADOOP-9092-branch-2.patch, HADOOP-9092-trunk-a.patch, HADOOP-9092-trunk.patch
Coverage fixing for package org.apache.hadoop.mapreduce.jobhistory
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
