[jira] [Created] (HADOOP-8521) Port StreamInputFormat to new Map Reduce API
madhukara phatak created HADOOP-8521: Summary: Port StreamInputFormat to new Map Reduce API Key: HADOOP-8521 URL: https://issues.apache.org/jira/browse/HADOOP-8521 Project: Hadoop Common Issue Type: Improvement Affects Versions: 0.23.0 Reporter: madhukara phatak Assignee: madhukara phatak As of now , hadoop streaming still used old Hadoop API. This JIRA ports it to the new M/R API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8522) ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used
Mike Percy created HADOOP-8522: -- Summary: ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used Key: HADOOP-8522 URL: https://issues.apache.org/jira/browse/HADOOP-8522 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 2.0.0-alpha, 1.0.3 Reporter: Mike Percy ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used. The issue is that finish() flushes the compressor buffer and writes the gzip CRC32 + data length trailer. After that, resetState() does not repeat the gzip header, but simply starts writing more deflate-compressed data. The resultant files are not readable by the Linux gunzip tool. ResetableGzipOutputStream should write valid multi-member gzip files. The gzip format is specified in [RFC 1952|https://tools.ietf.org/html/rfc1952]. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Build failed in Jenkins: Hadoop-Common-0.23-Build #290
See https://builds.apache.org/job/Hadoop-Common-0.23-Build/290/changes Changes: [bobby] svn merge -c 1352330. FIXES: MAPREDUCE-3889. job client tries to use /tasklog interface, but that doesn't exist anymore (Devaraj K via bobby) -- [...truncated 13294 lines...] Running org.apache.hadoop.fs.viewfs.TestViewFsWithAuthorityLocalFs Tests run: 42, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.072 sec Running org.apache.hadoop.fs.viewfs.TestViewFsLocalFs Tests run: 42, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.103 sec Running org.apache.hadoop.fs.TestGlobPattern Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.158 sec Running org.apache.hadoop.fs.TestS3_LocalFileContextURI Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.132 sec Running org.apache.hadoop.fs.TestLocalFSFileContextCreateMkdir Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.761 sec Running org.apache.hadoop.fs.TestHarFileSystem Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.343 sec Running org.apache.hadoop.fs.TestFileSystemCaching Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.773 sec Running org.apache.hadoop.fs.TestLocalFsFCStatistics Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.606 sec Running org.apache.hadoop.fs.TestHardLink Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.31 sec Running org.apache.hadoop.fs.TestCommandFormat Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.164 sec Running org.apache.hadoop.fs.TestLocal_S3FileContextURI Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.12 sec Running org.apache.hadoop.fs.TestLocalFileSystem Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.846 sec Running org.apache.hadoop.fs.TestFcLocalFsPermission Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.651 sec Running org.apache.hadoop.fs.TestListFiles Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.617 sec Running org.apache.hadoop.fs.TestPath Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.861 sec Running org.apache.hadoop.fs.kfs.TestKosmosFileSystem Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.613 sec Running org.apache.hadoop.fs.TestGlobExpander Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.112 sec Running org.apache.hadoop.fs.TestFilterFileSystem Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.644 sec Running org.apache.hadoop.fs.TestFcLocalFsUtil Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.587 sec Running org.apache.hadoop.fs.TestGetFileBlockLocations Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.817 sec Running org.apache.hadoop.fs.s3.TestInMemoryS3FileSystemContract Tests run: 29, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.858 sec Running org.apache.hadoop.fs.s3.TestINode Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.121 sec Running org.apache.hadoop.fs.s3.TestS3Credentials Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.206 sec Running org.apache.hadoop.fs.s3.TestS3FileSystem Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.23 sec Running org.apache.hadoop.fs.TestDU Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.273 sec Running org.apache.hadoop.record.TestBuffer Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.125 sec Running org.apache.hadoop.record.TestRecordVersioning Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.21 sec Running org.apache.hadoop.record.TestRecordIO Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.23 sec Running org.apache.hadoop.metrics2.source.TestJvmMetrics Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.45 sec Running org.apache.hadoop.metrics2.util.TestSampleStat Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.142 sec Running org.apache.hadoop.metrics2.util.TestMetricsCache Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.144 sec Running org.apache.hadoop.metrics2.lib.TestInterns Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.273 sec Running org.apache.hadoop.metrics2.lib.TestMetricsAnnotations Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.495 sec Running org.apache.hadoop.metrics2.lib.TestMutableMetrics Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.49 sec Running org.apache.hadoop.metrics2.lib.TestUniqNames Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.161 sec Running org.apache.hadoop.metrics2.lib.TestMetricsRegistry Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.448 sec Running org.apache.hadoop.metrics2.impl.TestMetricsCollectorImpl Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.3 sec Running
[jira] [Resolved] (HADOOP-8517) --config option does not work with Hadoop installation on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp resolved HADOOP-8517. - Resolution: Not A Problem --config option does not work with Hadoop installation on Windows - Key: HADOOP-8517 URL: https://issues.apache.org/jira/browse/HADOOP-8517 Project: Hadoop Common Issue Type: Bug Reporter: Trupti Dhavle I ran following command hadoop --config c:\directory\hadoop\conf fs -ls / I get following error for --config option Unrecognized option: --config Could not create the Java virtual machine. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8523) test-patch.sh is doesn't validate patches before building
Jack Dintruff created HADOOP-8523: - Summary: test-patch.sh is doesn't validate patches before building Key: HADOOP-8523 URL: https://issues.apache.org/jira/browse/HADOOP-8523 Project: Hadoop Common Issue Type: Bug Components: scripts Affects Versions: 0.23.1 Reporter: Jack Dintruff Priority: Trivial When running test-patch.sh with an invalid patch (not formatted properly) or one that doesn't compile, the script spends a lot of time building Hadoop before checking to see if the patch is invalid. It would help devs if it checked first just in case we run test-patch.sh with a bad patch file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HADOOP-8517) --config option does not work with Hadoop installation on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Trupti Dhavle reopened HADOOP-8517: --- --config is a switch given to export the HADOOP_CONF_DIR path in the command line. When I run this on non-windows environment, the command works- /Users/trupti/WORK/Software/Hadoop/hadoop-1.0.3/bin/hadoop --config /Users/trupti/WORK/Software/Hadoop/hadoop-1.0.3/conf dfs -ls / Found 3 items drwxr-xr-x - trupti supergroup 0 2012-06-12 14:09 /hdfsRegressionData drwxr-xr-x - trupti supergroup 0 2012-06-12 11:31 /tmp drwxr-xr-x - trupti supergroup 0 2012-06-20 10:12 /user /Users/trupti/WORK/Software/Hadoop/hadoop-1.0.3/bin/hadoop Usage: hadoop [--config confdir] COMMAND where COMMAND is one of: namenode -format format the DFS filesystem ... ... But the same switch is unavailable for Windows Hadoop. --config option does not work with Hadoop installation on Windows - Key: HADOOP-8517 URL: https://issues.apache.org/jira/browse/HADOOP-8517 Project: Hadoop Common Issue Type: Bug Reporter: Trupti Dhavle I ran following command hadoop --config c:\directory\hadoop\conf fs -ls / I get following error for --config option Unrecognized option: --config Could not create the Java virtual machine. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HADOOP-8486) Resource leak - Close the open resource handles (File handles) before throwing the exception from the SequenceFile constructor
[ https://issues.apache.org/jira/browse/HADOOP-8486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha reopened HADOOP-8486: This has caused build failure because Path.getUriPath is no longer available. [javac] c:\Users\bikas\Code\hdp\src\test\org\apache\hadoop\io\TestSequenceFile.java:59: cannot find symbol [javac] symbol : method getUriPath() [javac] location: class org.apache.hadoop.fs.Path [javac] File f = new File(nonSeqFile.getUriPath()); Resource leak - Close the open resource handles (File handles) before throwing the exception from the SequenceFile constructor -- Key: HADOOP-8486 URL: https://issues.apache.org/jira/browse/HADOOP-8486 Project: Hadoop Common Issue Type: Bug Components: fs, io Affects Versions: 1.0.2, 1-win Reporter: Kanna Karanam Assignee: Kanna Karanam Fix For: 1-win Attachments: HADOOP-8486-branch-1-win-(2).patch, HADOOP-8486-branch-1-win-(3).patch, HADOOP-8486-branch-1-win-(4).patch, HADOOP-8486-branch-1-win.patch I noticed this problem while I am working on porting HIVE to work on windows. Hive is attempting to create this class object to validate the file format and end up with resource leak. Because of this leak, we can’t move, rename or delete the files on windows when there is an open file handle whereas in UNIX we can perform all these operation with no issues even with open file handles. Please suggest me if you similar issues in any other places. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Mac OS-X apache hadoop installation issues
I too am getting the same errors when I am running the same command on Ubuntu.I have searched the whole internet but could not find a solution to it.Were you able to solve? If yes, then will you help me? I have started snatching my hairs!!!
[jira] [Created] (HADOOP-8525) Provide Improved Traceability for Configuration
Robert Joseph Evans created HADOOP-8525: --- Summary: Provide Improved Traceability for Configuration Key: HADOOP-8525 URL: https://issues.apache.org/jira/browse/HADOOP-8525 Project: Hadoop Common Issue Type: Improvement Reporter: Robert Joseph Evans Priority: Trivial Configuration provides basic traceability to see where a config setting came from, but once the configuration is written out that information is written to a comment in the XML and then lost the next time the configuration is read back in. It would really be great to be able to store a complete history of where the config came from in the XML, so that it can then be retrieved later for debugging. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Mac OS-X apache hadoop installation issues
Nikit, We had responded to this earlier with a possible solution (Use a _proper_ jar path being it.) See the response at http://search-hadoop.com/m/8ryfMcUvPQ1 On Fri, Jun 22, 2012 at 2:03 AM, Nikit Saraf nikit.sa...@gmail.com wrote: I too am getting the same errors when I am running the same command on Ubuntu.I have searched the whole internet but could not find a solution to it.Were you able to solve? If yes, then will you help me? I have started snatching my hairs!!! HTH! -- Harsh J
[jira] [Resolved] (HADOOP-8409) Fix TestCommandLineJobSubmission and TestGenericOptionsParser to work for windows
[ https://issues.apache.org/jira/browse/HADOOP-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha resolved HADOOP-8409. Resolution: Fixed Fix TestCommandLineJobSubmission and TestGenericOptionsParser to work for windows - Key: HADOOP-8409 URL: https://issues.apache.org/jira/browse/HADOOP-8409 Project: Hadoop Common Issue Type: Bug Components: fs, test, util Affects Versions: 1.0.0 Reporter: Ivan Mitic Assignee: Ivan Mitic Attachments: HADOOP-8409-branch-1-win.2.patch, HADOOP-8409-branch-1-win.3.patch, HADOOP-8409-branch-1-win.4.patch, HADOOP-8409-branch-1-win.patch Original Estimate: 168h Remaining Estimate: 168h There are multiple places in prod and test code where Windows paths are not handled properly. From a high level this could be summarized with: 1. Windows paths are not necessarily valid DFS paths (while Unix paths are) 2. Windows paths are not necessarily valid URIs (while Unix paths are) #1 causes a number of tests to fail because they implicitly assume that local paths are valid DFS paths (by extracting the DFS test path from for example test.build.data property) #2 causes issues when URIs are directly created on path strings passed in by the user -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Branch for Hadoop on Windows
Hi, branch-1-win was created to stage the work needed to make Hadoop run natively on Windows. There has been great progress on it that all of us can try out. As that work nears completion, I would like have a corresponding branch-trunk-win created to port the work to trunk. Thanks Bikas
Re: Branch for Hadoop on Windows
Sounds great, I'll create the branch for you. Thanks for the your efforts on this so far! On Thu, Jun 21, 2012 at 4:58 PM, Bikas Saha bi...@hortonworks.com wrote: Hi, branch-1-win was created to stage the work needed to make Hadoop run natively on Windows. There has been great progress on it that all of us can try out. As that work nears completion, I would like have a corresponding branch-trunk-win created to port the work to trunk. Thanks Bikas
