[jira] [Commented] (HADOOP-3450) Add tests to Local Directory Allocator for asserting their URI-returning capability
[
https://issues.apache.org/jira/browse/HADOOP-3450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13405018#comment-13405018
]
Hudson commented on HADOOP-3450:
Integrated in Hadoop-Hdfs-trunk #1093 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1093/])
HADOOP-3450. Add tests to Local Directory Allocator for asserting their
URI-returning capability. Contributed by Sho Shimauchi. (harsh) (Revision
1355789)
Result = FAILURE
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1355789
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalDirAllocator.java
Add tests to Local Directory Allocator for asserting their URI-returning
capability
---
Key: HADOOP-3450
URL: https://issues.apache.org/jira/browse/HADOOP-3450
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Affects Versions: 0.17.0
Reporter: Ari Rabkin
Assignee: Sho Shimauchi
Priority: Minor
Labels: newbie
Fix For: 2.0.1-alpha
Attachments: HADOOP-3450.txt
Original comment:
{quote}Local directory allocator returns a bare path, without a URI
specifier. This means that calling Path.getFileSystem will do the wrong
thing with the returned path. Should really stick a file:// in front.
Also it's test cases need to be improved to make sure this class works fine.
{quote}
Only the latter needed to be done (see below for discussion).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8538) CMake builds fail on ARM
[ https://issues.apache.org/jira/browse/HADOOP-8538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13405020#comment-13405020 ] Hudson commented on HADOOP-8538: Integrated in Hadoop-Hdfs-trunk #1093 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1093/]) HADOOP-8538. CMake builds fail on ARM. Contributed by Trevor Robinson (Revision 1356041) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1356041 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/CMakeLists.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/CMakeLists.txt CMake builds fail on ARM Key: HADOOP-8538 URL: https://issues.apache.org/jira/browse/HADOOP-8538 Project: Hadoop Common Issue Type: Bug Components: native Environment: gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) Reporter: Trevor Robinson Assignee: Trevor Robinson Fix For: 2.0.1-alpha Attachments: hadoop-cmake.patch CMake native builds fail with this error: cc1: error: unrecognized command line option '-m32' -m32 is only defined by GCC for x86, PowerPC, and SPARC. The following files specify -m32 when the JVM data model is 32-bit: hadoop-common-project/hadoop-common/src/CMakeLists.txt hadoop-hdfs-project/hadoop-hdfs/src/CMakeLists.txt hadoop-tools/hadoop-pipes/src/CMakeLists.txt hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt This is a partial regression of HDFS-1920. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6802) Remove FS_CLIENT_BUFFER_DIR_KEY = fs.client.buffer.dir from CommonConfigurationKeys.java (not used, deprecated)
[ https://issues.apache.org/jira/browse/HADOOP-6802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13405017#comment-13405017 ] Hudson commented on HADOOP-6802: Integrated in Hadoop-Hdfs-trunk #1093 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1093/]) HADOOP-6802. Remove FS_CLIENT_BUFFER_DIR_KEY = fs.client.buffer.dir from CommonConfigurationKeys.java (not used, deprecated). Contributed by Sho Shimauchi. (harsh) (Revision 1355784) Result = FAILURE harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1355784 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalDirAllocator.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalDirAllocator.java Remove FS_CLIENT_BUFFER_DIR_KEY = fs.client.buffer.dir from CommonConfigurationKeys.java (not used, deprecated) - Key: HADOOP-6802 URL: https://issues.apache.org/jira/browse/HADOOP-6802 Project: Hadoop Common Issue Type: Improvement Components: conf, fs Affects Versions: 0.23.0 Reporter: Erik Steffl Assignee: Sho Shimauchi Labels: newbie Fix For: 2.0.1-alpha Attachments: HADOOP-6802.txt, HADOOP-6802.txt In CommonConfigurationKeys.java: public static final String FS_CLIENT_BUFFER_DIR_KEY = fs.client.buffer.dir; The variable FS_CLIENT_BUFFER_DIR_KEY and string fs.client.buffer.dir are not used anywhere (Checked Hadoop Common, Hdfs and Mapred projects), it seems they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8521) Port StreamInputFormat to new Map Reduce API
[ https://issues.apache.org/jira/browse/HADOOP-8521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhukara phatak updated HADOOP-8521: - Status: Patch Available (was: Open) Port StreamInputFormat to new Map Reduce API Key: HADOOP-8521 URL: https://issues.apache.org/jira/browse/HADOOP-8521 Project: Hadoop Common Issue Type: Improvement Affects Versions: 0.23.0 Reporter: madhukara phatak Assignee: madhukara phatak Attachments: HADOOP-8521-1.patch, HADOOP-8521-2.patch, HADOOP-8521-3.patch, HADOOP-8521.patch As of now , hadoop streaming uses old Hadoop M/R API. This JIRA ports it to the new M/R API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8521) Port StreamInputFormat to new Map Reduce API
[ https://issues.apache.org/jira/browse/HADOOP-8521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13405052#comment-13405052 ] Hadoop QA commented on HADOOP-8521: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12534149/HADOOP-8521-3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. -1 javac. The applied patch generated 2073 javac compiler warnings (more than the trunk's current 2070 warnings). +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-tools/hadoop-streaming. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1161//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/1161//artifact/trunk/trunk/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1161//console This message is automatically generated. Port StreamInputFormat to new Map Reduce API Key: HADOOP-8521 URL: https://issues.apache.org/jira/browse/HADOOP-8521 Project: Hadoop Common Issue Type: Improvement Affects Versions: 0.23.0 Reporter: madhukara phatak Assignee: madhukara phatak Attachments: HADOOP-8521-1.patch, HADOOP-8521-2.patch, HADOOP-8521-3.patch, HADOOP-8521.patch As of now , hadoop streaming uses old Hadoop M/R API. This JIRA ports it to the new M/R API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8521) Port StreamInputFormat to new Map Reduce API
[ https://issues.apache.org/jira/browse/HADOOP-8521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13405093#comment-13405093 ] Robert Joseph Evans commented on HADOOP-8521: - Please take a look at the new javac warnings https://builds.apache.org/job/PreCommit-HADOOP-Build/1161/artifact/trunk/patchprocess/diffJavacWarnings.txt You don't necessarily need to fix them, just make sure that they are what you expect and are things that don't need to be fixed. Port StreamInputFormat to new Map Reduce API Key: HADOOP-8521 URL: https://issues.apache.org/jira/browse/HADOOP-8521 Project: Hadoop Common Issue Type: Improvement Affects Versions: 0.23.0 Reporter: madhukara phatak Assignee: madhukara phatak Attachments: HADOOP-8521-1.patch, HADOOP-8521-2.patch, HADOOP-8521-3.patch, HADOOP-8521.patch As of now , hadoop streaming uses old Hadoop M/R API. This JIRA ports it to the new M/R API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8521) Port StreamInputFormat to new Map Reduce API
[ https://issues.apache.org/jira/browse/HADOOP-8521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13405101#comment-13405101 ] madhukara phatak commented on HADOOP-8521: -- Hi Robert, These javac warning are resulted from use of Reflection API and old API deprecation. I think these warning should be fine since they don't effect actual functionality. Port StreamInputFormat to new Map Reduce API Key: HADOOP-8521 URL: https://issues.apache.org/jira/browse/HADOOP-8521 Project: Hadoop Common Issue Type: Improvement Affects Versions: 0.23.0 Reporter: madhukara phatak Assignee: madhukara phatak Attachments: HADOOP-8521-1.patch, HADOOP-8521-2.patch, HADOOP-8521-3.patch, HADOOP-8521.patch As of now , hadoop streaming uses old Hadoop M/R API. This JIRA ports it to the new M/R API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8543) Invalid pom.xml files on 0.23 branch
[
https://issues.apache.org/jira/browse/HADOOP-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13405105#comment-13405105
]
Robert Joseph Evans commented on HADOOP-8543:
-
branch-0.23 does not seem to build for me after applying the patch. I am using
RHEL 5, mvn 3.0.3, the sun JDK 1.6.0_26.
I get some classpath errors like
{noformat}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile (default-compile)
on project hadoop-hdfs: Compilation failure: Compilation failure:
[ERROR]
branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java:[105,63]
package org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos does not exist
{noformat}
Radim, did you see anything like this?
Invalid pom.xml files on 0.23 branch
Key: HADOOP-8543
URL: https://issues.apache.org/jira/browse/HADOOP-8543
Project: Hadoop Common
Issue Type: Bug
Components: build
Affects Versions: 0.23.3
Environment: FreeBSD 8.2, 64bit, Artifactory
Reporter: Radim Kolar
Labels: build
Fix For: 0.23.3
Attachments: hadoop-invalid-pom-023.txt
This is backport of HADOOP-8268 to 0.23 branch. It fixes invalid pom.xml
which allows them to be uploaded into artifactory maven repository management
and adds schema declarations which allows to use XML validating tools.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8543) Invalid pom.xml files on 0.23 branch
[
https://issues.apache.org/jira/browse/HADOOP-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13405111#comment-13405111
]
Robert Joseph Evans commented on HADOOP-8543:
-
Oh I see now hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml
{noformat}
$ diff --ignore-space-change hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml
../branch-2/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml
...
488c480
which cygpath gt; /dev/null
---
which cygpath 2gt; /dev/null
531c523
which cygpath gt; /dev/null
---
which cygpath 2gt; /dev/null
{noformat}
It looks like there was a mixup in the cygpath line 2 got changed to
but still with invalid XML.
Invalid pom.xml files on 0.23 branch
Key: HADOOP-8543
URL: https://issues.apache.org/jira/browse/HADOOP-8543
Project: Hadoop Common
Issue Type: Bug
Components: build
Affects Versions: 0.23.3
Environment: FreeBSD 8.2, 64bit, Artifactory
Reporter: Radim Kolar
Labels: build
Fix For: 0.23.3
Attachments: hadoop-invalid-pom-023.txt
This is backport of HADOOP-8268 to 0.23 branch. It fixes invalid pom.xml
which allows them to be uploaded into artifactory maven repository management
and adds schema declarations which allows to use XML validating tools.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8543) Invalid pom.xml files on 0.23 branch
[ https://issues.apache.org/jira/browse/HADOOP-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated HADOOP-8543: Status: Open (was: Patch Available) Invalid pom.xml files on 0.23 branch Key: HADOOP-8543 URL: https://issues.apache.org/jira/browse/HADOOP-8543 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 0.23.3 Environment: FreeBSD 8.2, 64bit, Artifactory Reporter: Radim Kolar Labels: build Fix For: 0.23.3 Attachments: hadoop-invalid-pom-023.txt This is backport of HADOOP-8268 to 0.23 branch. It fixes invalid pom.xml which allows them to be uploaded into artifactory maven repository management and adds schema declarations which allows to use XML validating tools. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8463) hadoop.security.auth_to_local needs a key definition and doc
[ https://issues.apache.org/jira/browse/HADOOP-8463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhukara phatak updated HADOOP-8463: - Attachment: HADOOP-8463-3.patch hadoop.security.auth_to_local needs a key definition and doc - Key: HADOOP-8463 URL: https://issues.apache.org/jira/browse/HADOOP-8463 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 2.0.0-alpha Reporter: Eli Collins Assignee: madhukara phatak Labels: newbie Attachments: HADOOP-8463-1.patch, HADOOP-8463-2.patch, HADOOP-8463-3.patch, HADOOP-8463.patch, HADOOP-8463.patch hadoop.security.auth_to_local should be defined in CommonConfigurationKeysPublic.java, and update the uses of the raw string in common and hdfs with the key. It's definition in core-site.xml should also be updated with a description. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Moved] (HADOOP-8547) Package hadoop-pipes examples/bin directory (again)
[ https://issues.apache.org/jira/browse/HADOOP-8547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins moved HDFS-3589 to HADOOP-8547: --- Key: HADOOP-8547 (was: HDFS-3589) Project: Hadoop Common (was: Hadoop HDFS) Package hadoop-pipes examples/bin directory (again) --- Key: HADOOP-8547 URL: https://issues.apache.org/jira/browse/HADOOP-8547 Project: Hadoop Common Issue Type: Bug Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: HDFS-3589.001.patch It looks like since MAPREDUCE-4267, we're no longer exporting the hadooppipes examples/bin directory to hadoop-dist as part of a mvn package build. This seems unintentional, so we should export those binaries again. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8547) Package hadoop-pipes examples/bin directory (again)
[ https://issues.apache.org/jira/browse/HADOOP-8547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-8547: - Attachment: HDFS-3589.001.patch Package hadoop-pipes examples/bin directory (again) --- Key: HADOOP-8547 URL: https://issues.apache.org/jira/browse/HADOOP-8547 Project: Hadoop Common Issue Type: Bug Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: HDFS-3589.001.patch It looks like since MAPREDUCE-4267, we're no longer exporting the hadooppipes examples/bin directory to hadoop-dist as part of a mvn package build. This seems unintentional, so we should export those binaries again. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8547) Package hadoop-pipes examples/bin directory (again)
[ https://issues.apache.org/jira/browse/HADOOP-8547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-8547: - Status: Patch Available (was: Open) Package hadoop-pipes examples/bin directory (again) --- Key: HADOOP-8547 URL: https://issues.apache.org/jira/browse/HADOOP-8547 Project: Hadoop Common Issue Type: Bug Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: HDFS-3589.001.patch It looks like since MAPREDUCE-4267, we're no longer exporting the hadooppipes examples/bin directory to hadoop-dist as part of a mvn package build. This seems unintentional, so we should export those binaries again. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8546) missing hdfs user guide and command pages from hadoop project page
[ https://issues.apache.org/jira/browse/HADOOP-8546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins resolved HADOOP-8546. - Resolution: Duplicate Per HDFS-3458 the forest docs needed to be ported to APT. missing hdfs user guide and command pages from hadoop project page Key: HADOOP-8546 URL: https://issues.apache.org/jira/browse/HADOOP-8546 Project: Hadoop Common Issue Type: Improvement Reporter: Jason Shih Priority: Trivial cant find the web master contact thus file issue in jira. two of the following links are missing from hadoop project page, maybe you can help restoring back them. thanks http://hadoop.apache.org/common/docs/current/hadoop-project-dist/hadoop-hdfs/hdfs_user_guide.html http://hadoop.apache.org/common/docs/current/hadoop-project-dist/hadoop-common/commands_manual.html Cheers, Jason -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8547) Package hadoop-pipes examples/bin directory (again)
[ https://issues.apache.org/jira/browse/HADOOP-8547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13405162#comment-13405162 ] Eli Collins commented on HADOOP-8547: - Why remove the installation of the headers? Otherwise looks good. Package hadoop-pipes examples/bin directory (again) --- Key: HADOOP-8547 URL: https://issues.apache.org/jira/browse/HADOOP-8547 Project: Hadoop Common Issue Type: Bug Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: HDFS-3589.001.patch It looks like since MAPREDUCE-4267, we're no longer exporting the hadooppipes examples/bin directory to hadoop-dist as part of a mvn package build. This seems unintentional, so we should export those binaries again. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8463) hadoop.security.auth_to_local needs a key definition and doc
[ https://issues.apache.org/jira/browse/HADOOP-8463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13405174#comment-13405174 ] Eli Collins commented on HADOOP-8463: - Looks great, +1 pending jenkins hadoop.security.auth_to_local needs a key definition and doc - Key: HADOOP-8463 URL: https://issues.apache.org/jira/browse/HADOOP-8463 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 2.0.0-alpha Reporter: Eli Collins Assignee: madhukara phatak Labels: newbie Attachments: HADOOP-8463-1.patch, HADOOP-8463-2.patch, HADOOP-8463-3.patch, HADOOP-8463.patch, HADOOP-8463.patch hadoop.security.auth_to_local should be defined in CommonConfigurationKeysPublic.java, and update the uses of the raw string in common and hdfs with the key. It's definition in core-site.xml should also be updated with a description. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8547) Package hadoop-pipes examples/bin directory (again)
[
https://issues.apache.org/jira/browse/HADOOP-8547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13405185#comment-13405185
]
Colin Patrick McCabe commented on HADOOP-8547:
--
The headers don't need to be installed because hadoop-tools.xml takes them
directly out of the source tree.
{code}
fileSet
directory../hadoop-pipes/src/main/native/utils/api/hadoop/directory
includes
include*.hh/include
/includes
outputDirectory/include/outputDirectory
/fileSet
{code}
{code}
cmccabe@keter:~/hadoop5 ls -l
hadoop-tools/hadoop-pipes/src/main/native/utils/api/hadoop/
total 12
-rw-r--r-- 1 cmccabe users 4514 Jul 2 10:34 SerialUtils.hh
-rw-r--r-- 1 cmccabe users 2441 Jul 2 10:34 StringUtils.hh
{code}
Another way to see that this is true is to realize that we never call make
install so all INSTALL sections are null and void anyway.
Package hadoop-pipes examples/bin directory (again)
---
Key: HADOOP-8547
URL: https://issues.apache.org/jira/browse/HADOOP-8547
Project: Hadoop Common
Issue Type: Bug
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Attachments: HDFS-3589.001.patch
It looks like since MAPREDUCE-4267, we're no longer exporting the hadooppipes
examples/bin directory to hadoop-dist as part of a mvn package build. This
seems unintentional, so we should export those binaries again.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8548) test-patch.sh shows an incorrect link in Jekins builds
Kihwal Lee created HADOOP-8548:
--
Summary: test-patch.sh shows an incorrect link in Jekins builds
Key: HADOOP-8548
URL: https://issues.apache.org/jira/browse/HADOOP-8548
Project: Hadoop Common
Issue Type: Bug
Components: build
Reporter: Kihwal Lee
Precommit builds show an incorrect link for javac warnings.
{noformat}
Javac warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2539//artifact/trunk/trunk/patchprocess/diffJavacWarnings.txt
{noformat}
Note that 'trunk' appears twice. Do we need $(basename $BASEDIR) in the
following? Other places don't have it.
{code:title=test-patch.sh}
JIRA_COMMENT_FOOTER=Javac warnings: $BUILD_URL/artifact/trunk/$(basename
$BASEDIR)/patchprocess/diffJavacWarnings.txt
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8548) test-patch.sh shows an incorrect link in Jekins builds
[
https://issues.apache.org/jira/browse/HADOOP-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HADOOP-8548:
---
Description:
Precommit builds show an incorrect link for javac warnings.
{noformat}
Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/
2539//artifact/trunk/trunk/patchprocess/diffJavacWarnings.txt
{noformat}
Note that 'trunk' appears twice. Do we need $(basename $BASEDIR) in the
following? Other places don't have it.
{code:title=test-patch.sh}
JIRA_COMMENT_FOOTER=Javac warnings: $BUILD_URL/artifact/trunk/
$(basename $BASEDIR)/patchprocess/diffJavacWarnings.txt
{code}
was:
Precommit builds show an incorrect link for javac warnings.
{noformat}
Javac warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2539//artifact/trunk/trunk/patchprocess/diffJavacWarnings.txt
{noformat}
Note that 'trunk' appears twice. Do we need $(basename $BASEDIR) in the
following? Other places don't have it.
{code:title=test-patch.sh}
JIRA_COMMENT_FOOTER=Javac warnings: $BUILD_URL/artifact/trunk/$(basename
$BASEDIR)/patchprocess/diffJavacWarnings.txt
{code}
test-patch.sh shows an incorrect link in Jekins builds
--
Key: HADOOP-8548
URL: https://issues.apache.org/jira/browse/HADOOP-8548
Project: Hadoop Common
Issue Type: Bug
Components: build
Reporter: Kihwal Lee
Precommit builds show an incorrect link for javac warnings.
{noformat}
Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/
2539//artifact/trunk/trunk/patchprocess/diffJavacWarnings.txt
{noformat}
Note that 'trunk' appears twice. Do we need $(basename $BASEDIR) in the
following? Other places don't have it.
{code:title=test-patch.sh}
JIRA_COMMENT_FOOTER=Javac warnings: $BUILD_URL/artifact/trunk/
$(basename $BASEDIR)/patchprocess/diffJavacWarnings.txt
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8548) test-patch.sh shows an incorrect link in Jekins builds
[
https://issues.apache.org/jira/browse/HADOOP-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HADOOP-8548:
---
Assignee: Kihwal Lee
Status: Patch Available (was: Open)
test-patch.sh shows an incorrect link in Jekins builds
--
Key: HADOOP-8548
URL: https://issues.apache.org/jira/browse/HADOOP-8548
Project: Hadoop Common
Issue Type: Bug
Components: build
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Attachments: hadoop-8548.patch
Precommit builds show an incorrect link for javac warnings.
{noformat}
Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/
2539//artifact/trunk/trunk/patchprocess/diffJavacWarnings.txt
{noformat}
Note that 'trunk' appears twice. Do we need $(basename $BASEDIR) in the
following? Other places don't have it.
{code:title=test-patch.sh}
JIRA_COMMENT_FOOTER=Javac warnings: $BUILD_URL/artifact/trunk/
$(basename $BASEDIR)/patchprocess/diffJavacWarnings.txt
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8548) test-patch.sh shows an incorrect link in Jekins builds
[
https://issues.apache.org/jira/browse/HADOOP-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HADOOP-8548:
---
Attachment: hadoop-8548.patch
test-patch.sh shows an incorrect link in Jekins builds
--
Key: HADOOP-8548
URL: https://issues.apache.org/jira/browse/HADOOP-8548
Project: Hadoop Common
Issue Type: Bug
Components: build
Reporter: Kihwal Lee
Attachments: hadoop-8548.patch
Precommit builds show an incorrect link for javac warnings.
{noformat}
Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/
2539//artifact/trunk/trunk/patchprocess/diffJavacWarnings.txt
{noformat}
Note that 'trunk' appears twice. Do we need $(basename $BASEDIR) in the
following? Other places don't have it.
{code:title=test-patch.sh}
JIRA_COMMENT_FOOTER=Javac warnings: $BUILD_URL/artifact/trunk/
$(basename $BASEDIR)/patchprocess/diffJavacWarnings.txt
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8547) Package hadoop-pipes examples/bin directory (again)
[ https://issues.apache.org/jira/browse/HADOOP-8547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13405216#comment-13405216 ] Eli Collins commented on HADOOP-8547: - Ah, great. +1 pending jenkins Package hadoop-pipes examples/bin directory (again) --- Key: HADOOP-8547 URL: https://issues.apache.org/jira/browse/HADOOP-8547 Project: Hadoop Common Issue Type: Bug Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: HDFS-3589.001.patch It looks like since MAPREDUCE-4267, we're no longer exporting the hadooppipes examples/bin directory to hadoop-dist as part of a mvn package build. This seems unintentional, so we should export those binaries again. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8463) hadoop.security.auth_to_local needs a key definition and doc
[ https://issues.apache.org/jira/browse/HADOOP-8463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13405219#comment-13405219 ] Hadoop QA commented on HADOOP-8463: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12534292/HADOOP-8463-3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.fs.viewfs.TestViewFsTrash +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1162//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1162//console This message is automatically generated. hadoop.security.auth_to_local needs a key definition and doc - Key: HADOOP-8463 URL: https://issues.apache.org/jira/browse/HADOOP-8463 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 2.0.0-alpha Reporter: Eli Collins Assignee: madhukara phatak Labels: newbie Attachments: HADOOP-8463-1.patch, HADOOP-8463-2.patch, HADOOP-8463-3.patch, HADOOP-8463.patch, HADOOP-8463.patch hadoop.security.auth_to_local should be defined in CommonConfigurationKeysPublic.java, and update the uses of the raw string in common and hdfs with the key. It's definition in core-site.xml should also be updated with a description. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8548) test-patch.sh shows an incorrect link in Jekins builds
[
https://issues.apache.org/jira/browse/HADOOP-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated HADOOP-8548:
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Thanks for the patch Kihwal, and thanks for fixing my bug :). +1 I put this
into trunk.
test-patch.sh shows an incorrect link in Jekins builds
--
Key: HADOOP-8548
URL: https://issues.apache.org/jira/browse/HADOOP-8548
Project: Hadoop Common
Issue Type: Bug
Components: build
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Fix For: 3.0.0
Attachments: hadoop-8548.patch
Precommit builds show an incorrect link for javac warnings.
{noformat}
Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/
2539//artifact/trunk/trunk/patchprocess/diffJavacWarnings.txt
{noformat}
Note that 'trunk' appears twice. Do we need $(basename $BASEDIR) in the
following? Other places don't have it.
{code:title=test-patch.sh}
JIRA_COMMENT_FOOTER=Javac warnings: $BUILD_URL/artifact/trunk/
$(basename $BASEDIR)/patchprocess/diffJavacWarnings.txt
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8533) Remove Parallel Call in IPC
[ https://issues.apache.org/jira/browse/HADOOP-8533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13405383#comment-13405383 ] Suresh Srinivas commented on HADOOP-8533: - +1 for the patch. Remove Parallel Call in IPC --- Key: HADOOP-8533 URL: https://issues.apache.org/jira/browse/HADOOP-8533 Project: Hadoop Common Issue Type: Bug Components: ipc Reporter: Suresh Srinivas Assignee: Brandon Li Fix For: 3.0.0 Attachments: HADOOP-8533.patch From what I know, I do not think any one uses Parallel Call. I also think it is not tested very well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8533) Remove Parallel Call in IPC
[ https://issues.apache.org/jira/browse/HADOOP-8533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HADOOP-8533: Description: From what I know, I do not think anyone uses Parallel Call. I also think it is not tested very well. (was: From what I know, I do not think any one uses Parallel Call. I also think it is not tested very well.) Affects Version/s: 3.0.0 1.0.0 2.0.0-alpha Issue Type: Improvement (was: Bug) Remove Parallel Call in IPC --- Key: HADOOP-8533 URL: https://issues.apache.org/jira/browse/HADOOP-8533 Project: Hadoop Common Issue Type: Improvement Components: ipc Affects Versions: 1.0.0, 2.0.0-alpha, 3.0.0 Reporter: Suresh Srinivas Assignee: Brandon Li Fix For: 3.0.0 Attachments: HADOOP-8533.patch From what I know, I do not think anyone uses Parallel Call. I also think it is not tested very well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8533) Remove Parallel Call in IPC
[ https://issues.apache.org/jira/browse/HADOOP-8533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HADOOP-8533: Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I committed the patch. Thank you Brandon. Remove Parallel Call in IPC --- Key: HADOOP-8533 URL: https://issues.apache.org/jira/browse/HADOOP-8533 Project: Hadoop Common Issue Type: Improvement Components: ipc Affects Versions: 1.0.0, 2.0.0-alpha, 3.0.0 Reporter: Suresh Srinivas Assignee: Brandon Li Fix For: 3.0.0 Attachments: HADOOP-8533.patch From what I know, I do not think anyone uses Parallel Call. I also think it is not tested very well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8533) Remove Parallel Call in IPC
[ https://issues.apache.org/jira/browse/HADOOP-8533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13405411#comment-13405411 ] Eli Collins commented on HADOOP-8533: - Hey Suresh, how about merging this to branch-2? Helps keep the diffs inline and as you point out is not used or well tested. Remove Parallel Call in IPC --- Key: HADOOP-8533 URL: https://issues.apache.org/jira/browse/HADOOP-8533 Project: Hadoop Common Issue Type: Improvement Components: ipc Affects Versions: 1.0.0, 2.0.0-alpha, 3.0.0 Reporter: Suresh Srinivas Assignee: Brandon Li Fix For: 3.0.0 Attachments: HADOOP-8533.patch From what I know, I do not think anyone uses Parallel Call. I also think it is not tested very well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8533) Remove Parallel Call in IPC
[ https://issues.apache.org/jira/browse/HADOOP-8533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HADOOP-8533: Fix Version/s: 2.0.1-alpha Release Note: Merged the change to branch-2 Remove Parallel Call in IPC --- Key: HADOOP-8533 URL: https://issues.apache.org/jira/browse/HADOOP-8533 Project: Hadoop Common Issue Type: Improvement Components: ipc Affects Versions: 1.0.0, 2.0.0-alpha, 3.0.0 Reporter: Suresh Srinivas Assignee: Brandon Li Fix For: 2.0.1-alpha, 3.0.0 Attachments: HADOOP-8533.patch From what I know, I do not think anyone uses Parallel Call. I also think it is not tested very well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8541) Better high-percentile latency metrics
[ https://issues.apache.org/jira/browse/HADOOP-8541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HADOOP-8541: Attachment: hadoop-8541-1.patch First attempt at a patch. I added a new MutableQuantiles class that tracks hardcoded quantiles, and rolls over at the specified interval. The test cases are pretty thorough, but I'm going to work on hooking this into either HDFS or HBase to make sure the API feels right. Better high-percentile latency metrics -- Key: HADOOP-8541 URL: https://issues.apache.org/jira/browse/HADOOP-8541 Project: Hadoop Common Issue Type: Improvement Components: metrics Affects Versions: 2.0.0-alpha Reporter: Andrew Wang Assignee: Andrew Wang Fix For: 2.0.1-alpha Attachments: hadoop-8541-1.patch Based on discussion in HBASE-6261 and with some HDFS devs, I'd like to make better high-percentile latency metrics a part of hadoop-common. I've already got a working implementation of [1], an efficient algorithm for estimating quantiles on a stream of values. It allows you to specify arbitrary quantiles to track (e.g. 50th, 75th, 90th, 95th, 99th), along with tight error bounds. This estimator can be snapshotted and reset periodically to get a feel for how these percentiles are changing over time. I propose creating a new MutableQuantiles class that does this. [1] isn't completely without overhead (~1MB memory for reasonably sized windows), which is why I hesitate to add it to the existing MutableStat class. [1] Cormode, Korn, Muthukrishnan, and Srivastava. Effective Computation of Biased Quantiles over Data Streams in ICDE 2005. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8534) Some tests leave a config file open causing failure on windows
[
https://issues.apache.org/jira/browse/HADOOP-8534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sanjay Radia updated HADOOP-8534:
-
Summary: Some tests leave a config file open causing failure on windows
(was: TestQueueManagerForJobKillAndJobPriority and
TestQueueManagerForJobKillAndNonDefaultQueue fail on Windows)
Some tests leave a config file open causing failure on windows
--
Key: HADOOP-8534
URL: https://issues.apache.org/jira/browse/HADOOP-8534
Project: Hadoop Common
Issue Type: Bug
Components: conf
Affects Versions: 1.0.0
Reporter: Ivan Mitic
Assignee: Ivan Mitic
Attachments: HADOOP-8534-branch-1-win_Parser(2).patch,
HADOOP-8534-branch-1-win_Parser.patch
Java xml parser keeps file locked after SAXException, causing the following
tests to fail:
- TestQueueManagerForJobKillAndJobPriority
- TestQueueManagerForJobKillAndNonDefaultQueue
{{TestQueueManagerForJobKillAndJobPriority#testQueueAclRefreshWithInvalidConfFile()}}
is creating a temp config file with incorrect syntax. Later, the test tries
to delete/cleanup this file and this operation fails on Windows (as the file
is still open). From this point on, all subsequent tests fail because they
try to use the incorrect config file.
Forum references on the problem and the fix:
http://www.linuxquestions.org/questions/programming-9/java-xml-parser-keeps-file-locked-after-saxexception-768613/
https://forums.oracle.com/forums/thread.jspa?threadID=2046505start=0tstart=0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8534) Some tests leave a config file open causing failure on windows
[
https://issues.apache.org/jira/browse/HADOOP-8534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sanjay Radia resolved HADOOP-8534.
--
Resolution: Fixed
Committed to Hadoop-1 windows branch. Thanks Ivan.
Some tests leave a config file open causing failure on windows
--
Key: HADOOP-8534
URL: https://issues.apache.org/jira/browse/HADOOP-8534
Project: Hadoop Common
Issue Type: Bug
Components: conf
Affects Versions: 1.0.0
Reporter: Ivan Mitic
Assignee: Ivan Mitic
Attachments: HADOOP-8534-branch-1-win_Parser(2).patch,
HADOOP-8534-branch-1-win_Parser.patch
Java xml parser keeps file locked after SAXException, causing the following
tests to fail:
- TestQueueManagerForJobKillAndJobPriority
- TestQueueManagerForJobKillAndNonDefaultQueue
{{TestQueueManagerForJobKillAndJobPriority#testQueueAclRefreshWithInvalidConfFile()}}
is creating a temp config file with incorrect syntax. Later, the test tries
to delete/cleanup this file and this operation fails on Windows (as the file
is still open). From this point on, all subsequent tests fail because they
try to use the incorrect config file.
Forum references on the problem and the fix:
http://www.linuxquestions.org/questions/programming-9/java-xml-parser-keeps-file-locked-after-saxexception-768613/
https://forums.oracle.com/forums/thread.jspa?threadID=2046505start=0tstart=0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8148) Zero-copy ByteBuffer-based compressor / decompressor API
[ https://issues.apache.org/jira/browse/HADOOP-8148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Broberg updated HADOOP-8148: Attachment: (was: zerocopyifc.tgz) Zero-copy ByteBuffer-based compressor / decompressor API Key: HADOOP-8148 URL: https://issues.apache.org/jira/browse/HADOOP-8148 Project: Hadoop Common Issue Type: New Feature Components: io, performance Reporter: Tim Broberg Assignee: Tim Broberg Attachments: hadoop-8148.patch, hadoop8148.patch Per Todd Lipcon's comment in HDFS-2834, Whenever a native decompression codec is being used, ... we generally have the following copies: 1) Socket - DirectByteBuffer (in SocketChannel implementation) 2) DirectByteBuffer - byte[] (in SocketInputStream) 3) byte[] - Native buffer (set up for decompression) 4*) decompression to a different native buffer (not really a copy - decompression necessarily rewrites) 5) native buffer - byte[] with the proposed improvement we can hopefully eliminate #2,#3 for all applications, and #2,#3,and #5 for libhdfs. The interfaces in the attached patch attempt to address: A - Compression and decompression based on ByteBuffers (HDFS-2834) B - Zero-copy compression and decompression (HDFS-3051) C - Provide the caller a way to know how the max space required to hold compressed output. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8148) Zero-copy ByteBuffer-based compressor / decompressor API
[ https://issues.apache.org/jira/browse/HADOOP-8148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Broberg updated HADOOP-8148: Attachment: zerocopyifc.tgz 2nd try - proposed zero copy compression interfaces Zero-copy ByteBuffer-based compressor / decompressor API Key: HADOOP-8148 URL: https://issues.apache.org/jira/browse/HADOOP-8148 Project: Hadoop Common Issue Type: New Feature Components: io, performance Reporter: Tim Broberg Assignee: Tim Broberg Attachments: hadoop-8148.patch, hadoop8148.patch, zerocopyifc.tgz Per Todd Lipcon's comment in HDFS-2834, Whenever a native decompression codec is being used, ... we generally have the following copies: 1) Socket - DirectByteBuffer (in SocketChannel implementation) 2) DirectByteBuffer - byte[] (in SocketInputStream) 3) byte[] - Native buffer (set up for decompression) 4*) decompression to a different native buffer (not really a copy - decompression necessarily rewrites) 5) native buffer - byte[] with the proposed improvement we can hopefully eliminate #2,#3 for all applications, and #2,#3,and #5 for libhdfs. The interfaces in the attached patch attempt to address: A - Compression and decompression based on ByteBuffers (HDFS-2834) B - Zero-copy compression and decompression (HDFS-3051) C - Provide the caller a way to know how the max space required to hold compressed output. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8148) Zero-copy ByteBuffer-based compressor / decompressor API
[ https://issues.apache.org/jira/browse/HADOOP-8148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13405492#comment-13405492 ] Tim Broberg commented on HADOOP-8148: - Ok, Owen. Sorry not to get back sooner. Last week was crazy. Non-empty tarball with my proposal is attached. Comments on yours are below, keeping in mind that I'm somewhat of a Java neophyte. If I say something stupid, please try to be gentle: 1 - By declaring an abstract stream class in the codec, you are forcing implementations to extend those classes, which means they can't reuse any of the existing compression stream code. What benefits do you see coming from this that warrant the tradeoff and/or what am I missing? 2 - What are the benefits of creating a separate codec / factory class vs just returning a compression stream and checking whether it implements ZeroCopywhatever? 3 - The read(ByteBuffer) definition just fundamentally doesn't work for my particular use case where there is a significant latency on reads. If the caller provides the ByteBuffer, then the read() function has to wait for the latency of the entire operation or else read() has to perform a copy from his own pool of completed buffers, which defeats the whole point of the exercise. This is why I felt I needed ByteBuffer readBuffer() in my interface. 4 - Am I seeing compress and decompress functions at the codec level? I guess you're trying to hide the complex setInput / compress interface of the existing compressor? I guess you're saying we want the compressors to still be available in non-stream form, but you want to clean up the interface? 5 - What about the codec pool? Do you see that disappearing? You mentioned channels. One alternative approach would be to use the new Java 7 Asynchronous channels, but making compression require Java 7 would have some pretty broad implications that I don't expect we're ready to deal with. Also, this makes the caller deal with the asynchronicity instead of having the stream just read ahead and return buffers on command. Ideally, it would be great to see an interface that doesn't require the caller to deal with pooling the buffers below his level in the stream, but I haven't found a way to do this that doesn't require copies for streams that implement any kind of read ahead. I'm sure at present I'm firmly in the minority in caring about multithreaded decompressors. I've tried to keep the overhead innocuous such that it's worth the trouble. Any ideas you have on addressing the problem more efficiently would be most welcome, but read(ByteBuffer) doesn't do it. Zero-copy ByteBuffer-based compressor / decompressor API Key: HADOOP-8148 URL: https://issues.apache.org/jira/browse/HADOOP-8148 Project: Hadoop Common Issue Type: New Feature Components: io, performance Reporter: Tim Broberg Assignee: Tim Broberg Attachments: hadoop-8148.patch, hadoop8148.patch, zerocopyifc.tgz Per Todd Lipcon's comment in HDFS-2834, Whenever a native decompression codec is being used, ... we generally have the following copies: 1) Socket - DirectByteBuffer (in SocketChannel implementation) 2) DirectByteBuffer - byte[] (in SocketInputStream) 3) byte[] - Native buffer (set up for decompression) 4*) decompression to a different native buffer (not really a copy - decompression necessarily rewrites) 5) native buffer - byte[] with the proposed improvement we can hopefully eliminate #2,#3 for all applications, and #2,#3,and #5 for libhdfs. The interfaces in the attached patch attempt to address: A - Compression and decompression based on ByteBuffers (HDFS-2834) B - Zero-copy compression and decompression (HDFS-3051) C - Provide the caller a way to know how the max space required to hold compressed output. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8547) Package hadoop-pipes examples/bin directory (again)
[ https://issues.apache.org/jira/browse/HADOOP-8547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13405499#comment-13405499 ] Hadoop QA commented on HADOOP-8547: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12534296/HDFS-3589.001.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-assemblies. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1163//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1163//console This message is automatically generated. Package hadoop-pipes examples/bin directory (again) --- Key: HADOOP-8547 URL: https://issues.apache.org/jira/browse/HADOOP-8547 Project: Hadoop Common Issue Type: Bug Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: HDFS-3589.001.patch It looks like since MAPREDUCE-4267, we're no longer exporting the hadooppipes examples/bin directory to hadoop-dist as part of a mvn package build. This seems unintentional, so we should export those binaries again. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8472) Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
[ https://issues.apache.org/jira/browse/HADOOP-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13405596#comment-13405596 ] Tsz Wo (Nicholas), SZE commented on HADOOP-8472: Hi Junping, It seems that all calls to TopologyResolver are in BlockPlacementPolicyWithNodeGroup and have withNodeGroupLayer == true. So is it possible to move the methods in TopologyResolver to BlockPlacementPolicyWithNodeGroup? Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology - Key: HADOOP-8472 URL: https://issues.apache.org/jira/browse/HADOOP-8472 Project: Hadoop Common Issue Type: Sub-task Components: ha, io Affects Versions: 1.0.0, 2.0.0-alpha Reporter: Junping Du Assignee: Junping Du Attachments: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v2.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v3.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v4.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology. The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are: 1. The 3rd replica will be off node group of the 2nd replica 2. If there is no local node available, the 1st replica will be placed on a node in the local node group. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8547) Package hadoop-pipes examples/bin directory (again)
[ https://issues.apache.org/jira/browse/HADOOP-8547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13405598#comment-13405598 ] Colin Patrick McCabe commented on HADOOP-8547: -- patch doesn't require new tests because it's not a code change, just a packaging change Package hadoop-pipes examples/bin directory (again) --- Key: HADOOP-8547 URL: https://issues.apache.org/jira/browse/HADOOP-8547 Project: Hadoop Common Issue Type: Bug Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: HDFS-3589.001.patch It looks like since MAPREDUCE-4267, we're no longer exporting the hadooppipes examples/bin directory to hadoop-dist as part of a mvn package build. This seems unintentional, so we should export those binaries again. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8472) Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
[ https://issues.apache.org/jira/browse/HADOOP-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13405608#comment-13405608 ] Junping Du commented on HADOOP-8472: Hey Nicholas, Some mapreduce patch will use this method too. Please refer patch in MAPREDUCE-4310. Any suggestions? Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology - Key: HADOOP-8472 URL: https://issues.apache.org/jira/browse/HADOOP-8472 Project: Hadoop Common Issue Type: Sub-task Components: ha, io Affects Versions: 1.0.0, 2.0.0-alpha Reporter: Junping Du Assignee: Junping Du Attachments: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v2.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v3.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v4.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology. The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are: 1. The 3rd replica will be off node group of the 2nd replica 2. If there is no local node available, the 1st replica will be placed on a node in the local node group. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8472) Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
[ https://issues.apache.org/jira/browse/HADOOP-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13405618#comment-13405618 ] Tsz Wo (Nicholas), SZE commented on HADOOP-8472: How about move the TopologyResolver methods to RackResolver since common and hdfs do not need them? Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology - Key: HADOOP-8472 URL: https://issues.apache.org/jira/browse/HADOOP-8472 Project: Hadoop Common Issue Type: Sub-task Components: ha, io Affects Versions: 1.0.0, 2.0.0-alpha Reporter: Junping Du Assignee: Junping Du Attachments: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v2.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v3.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v4.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology. The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are: 1. The 3rd replica will be off node group of the 2nd replica 2. If there is no local node available, the 1st replica will be placed on a node in the local node group. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8472) Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
[ https://issues.apache.org/jira/browse/HADOOP-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13405636#comment-13405636 ] Junping Du commented on HADOOP-8472: As your previous comments, TopologyResolver is called in BlockPlacementPolicyWithNodeGroup which is in hdfs. If we don't like a separated class to handle this, how about reuse NetworkTopology to handle the methods that currently in TopologyResolver? I think it should be something in common. Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology - Key: HADOOP-8472 URL: https://issues.apache.org/jira/browse/HADOOP-8472 Project: Hadoop Common Issue Type: Sub-task Components: ha, io Affects Versions: 1.0.0, 2.0.0-alpha Reporter: Junping Du Assignee: Junping Du Attachments: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v2.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v3.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v4.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology. The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are: 1. The 3rd replica will be off node group of the 2nd replica 2. If there is no local node available, the 1st replica will be placed on a node in the local node group. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
