[jira] [Created] (MAPREDUCE-4981) WordMean, WordMedian, WordStandardDeviation missing from ExamplesDriver
Plamen Jeliazkov created MAPREDUCE-4981: --- Summary: WordMean, WordMedian, WordStandardDeviation missing from ExamplesDriver Key: MAPREDUCE-4981 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4981 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0, 2.0.3-alpha Reporter: Plamen Jeliazkov Assignee: Plamen Jeliazkov Priority: Minor Fix For: 3.0.0, 2.0.3-alpha https://reviews.apache.org/r/1091/ introduced 3 new MapReduce examples, but they were never added to the ExamplesDriver. This JIRA proposes to add them to the ExamplesDriver. I have ran them myself and can confirm the examples still work as intended. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4985) TestDFSIO supports compression but usages doesn't reflect
Plamen Jeliazkov created MAPREDUCE-4985: --- Summary: TestDFSIO supports compression but usages doesn't reflect Key: MAPREDUCE-4985 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4985 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0 Reporter: Plamen Jeliazkov Assignee: Plamen Jeliazkov Priority: Trivial Fix For: 3.0.0 https://issues.apache.org/jira/browse/MAPREDUCE-2786 introduced the ability to use a compression codec during TestDFSIO. However, the -compression parameter was never introduced to the usages printout. This is a trivial patch to reveal the parameter to end users. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2786) TestDFSIO should also test compression reading/writing from command-line.
TestDFSIO should also test compression reading/writing from command-line. - Key: MAPREDUCE-2786 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2786 Project: Hadoop Map/Reduce Issue Type: Improvement Components: benchmarks Reporter: Plamen Jeliazkov Priority: Minor Fix For: 0.22.0 After running into trouble dealing with the config files I thought it might be easier to simply alter the code of TestDFSIO to accept any compression codec and allow testing for compression by a command line argument instead of having to change the config file everytime. Something like -compression would do. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: Addition of 3 new examples -- WordMean, WordMedian, and WordStandardDeviation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1091/ --- (Updated 2011-07-21 17:00:31.243531) Review request for hadoop-mapreduce. Changes --- FindBugs and javac warnings fixes. Summary --- Looking to add 3 new examples -- they can be added via the diff attached; some changes to the ExamplesDriver.java might be required however these files do work alone as well. I will also be attaching JUnit tests for these examples. I will post another review request for those. Diffs (updated) - http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/WordMean.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/WordMedian.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/WordStandardDeviation.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/test/mapred/org/apache/hadoop/examples/TestWordMean.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/test/mapred/org/apache/hadoop/examples/TestWordMedian.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/test/mapred/org/apache/hadoop/examples/TestWordStandardDeviation.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/test/mapred/org/apache/hadoop/examples/input/ShakespeareDream.txt PRE-CREATION Diff: https://reviews.apache.org/r/1091/diff Testing --- JUnit tests added -- reduced to 3 tests that all read from an input folder. Thanks, Plamen
Re: Review Request: Addition of 3 new examples -- WordMean, WordMedian, and WordStandardDeviation
On 2011-07-19 14:58:10, Harsh J wrote: http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/WordMedian.java, line 185 https://reviews.apache.org/r/1091/diff/4/?file=26229#file26229line185 If this class is public, use its .class.getCanonicalName() method directly via an import itself? Also, I though there were public counter enum groups available. Look for those instead of fetching w/ strings, and if they don't exist then this is fine. Plamen Jeliazkov wrote: I just removed this entirely. I am using the .findCounter(Str,Str) method directly instead. I realized I cannot just do that -- I could not find a stable way to use the public counter enum groups to get the MAP_OUTPUT_RECORDS. I know it was supposed to be fixed in 0.21.0, but I still feel like it is not easily accessed. For completion I have decided to just use your first idea with the .class.getCanonicalName() method. - Plamen --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1091/#review1109 --- On 2011-07-18 23:26:12, Plamen Jeliazkov wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1091/ --- (Updated 2011-07-18 23:26:12) Review request for hadoop-mapreduce. Summary --- Looking to add 3 new examples -- they can be added via the diff attached; some changes to the ExamplesDriver.java might be required however these files do work alone as well. I will also be attaching JUnit tests for these examples. I will post another review request for those. Diffs - http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/test/mapred/org/apache/hadoop/examples/TestWordStandardDeviation.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/test/mapred/org/apache/hadoop/examples/input/shaks12.txt PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/WordMedian.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/WordStandardDeviation.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/test/mapred/org/apache/hadoop/examples/TestWordMean.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/test/mapred/org/apache/hadoop/examples/TestWordMedian.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/WordMean.java PRE-CREATION Diff: https://reviews.apache.org/r/1091/diff Testing --- JUnit tests added -- reduced to 3 tests that all read from an input folder. Thanks, Plamen
Re: Review Request: Addition of 3 new examples -- WordMean, WordMedian, and WordStandardDeviation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1091/ --- (Updated 2011-07-21 01:30:49.709209) Review request for hadoop-mapreduce. Changes --- -Took many of the comments left by Harsh J to heart and followed through. Uses the ToolRunner now to init and run a job. Summary --- Looking to add 3 new examples -- they can be added via the diff attached; some changes to the ExamplesDriver.java might be required however these files do work alone as well. I will also be attaching JUnit tests for these examples. I will post another review request for those. Diffs (updated) - http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/WordMean.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/WordMedian.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/WordStandardDeviation.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/test/mapred/org/apache/hadoop/examples/TestWordMean.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/test/mapred/org/apache/hadoop/examples/TestWordMedian.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/test/mapred/org/apache/hadoop/examples/TestWordStandardDeviation.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/test/mapred/org/apache/hadoop/examples/input/ShakespeareDream.txt PRE-CREATION Diff: https://reviews.apache.org/r/1091/diff Testing --- JUnit tests added -- reduced to 3 tests that all read from an input folder. Thanks, Plamen
Re: Review Request: Addition of 3 new examples -- WordMean, WordMedian, and WordStandardDeviation
On 2011-07-19 14:58:10, Harsh J wrote: http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/WordMean.java, line 39 https://reviews.apache.org/r/1091/diff/4/?file=26228#file26228line39 Minor nits: Please remove all the extra whitespaces from empty lines and empty comment lines. If you look at your colorized git diff output, or the diff on reviewboard, you should be able to see where all these are present in bright red :) Yeah, just took a quick Code Style change in Eclipse and some Ctrl-Shift-F magic. Next patch will definitely have all those removed. On 2011-07-19 14:58:10, Harsh J wrote: http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/WordMean.java, line 146 https://reviews.apache.org/r/1091/diff/4/?file=26228#file26228line146 This shouldn't happen, and ought to be ensured by the test cases. I'd say its unnecessary here. The IOException handler doesn't add much value to the exception either, and can perhaps be avoided for a general throws IOException method? You make a good point -- I guess I have a habit of always using a try and catch together, never a try and finally. But I too felt a little unsure at first about the try/catch so it is nice to have some input. Thanks; I will remove both of them. :) On 2011-07-19 14:58:10, Harsh J wrote: http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/WordMedian.java, line 113 https://reviews.apache.org/r/1091/diff/4/?file=26229#file26229line113 If a file does not exist, perhaps its best not to return a wrong value but to fire away an exception? Yes, agreed. Fixed. On 2011-07-19 14:58:10, Harsh J wrote: http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/WordMedian.java, line 84 https://reviews.apache.org/r/1091/diff/4/?file=26229#file26229line84 Sum can too be made a member var with resets, and also the previous comment applies to Reducer's context write as well. You can utilize a single IntWritable stmt with set(int)-ing it every turn. Done. I have made sum var a private member inside of the Reducer. On 2011-07-19 14:58:10, Harsh J wrote: http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/WordMedian.java, line 185 https://reviews.apache.org/r/1091/diff/4/?file=26229#file26229line185 If this class is public, use its .class.getCanonicalName() method directly via an import itself? Also, I though there were public counter enum groups available. Look for those instead of fetching w/ strings, and if they don't exist then this is fine. I just removed this entirely. I am using the .findCounter(Str,Str) method directly instead. On 2011-07-19 14:58:10, Harsh J wrote: http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/WordMean.java, line 161 https://reviews.apache.org/r/1091/diff/4/?file=26228#file26228line161 Minor nit again, but you could also do GenericOptionsParser(args) first, and then extract the configuration out of it as GenericOptionsParser.getConfiguration(). This, cause you don't seem to be tweaking the original conf instance really. Again, minor nit, feel free to ignore :-) Yes I suppose it can be ignored but after messing around I feel like the change causes it to be a little more human readable than the original. Therefore, I will keep your change. Thanks. :) On 2011-07-19 14:58:10, Harsh J wrote: http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/WordMedian.java, line 162 https://reviews.apache.org/r/1091/diff/4/?file=26229#file26229line162 Also, instead of directly using GenericOptionsParser, have you considered writing it with the Tool class framework? That's how we usually recommend writing submission jars. I have just begun shifting over -- the number of imports we will save on the unit tests is well worth this change. Thank you! - Plamen --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1091/#review1109 --- On 2011-07-18 23:26:12, Plamen Jeliazkov wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1091/ --- (Updated 2011-07-18 23:26:12) Review request for hadoop-mapreduce. Summary --- Looking to add 3 new examples
Re: Review Request: Addition of 3 new examples -- WordMean, WordMedian, and WordStandardDeviation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1091/ --- (Updated 2011-07-18 23:26:12.317246) Review request for hadoop-mapreduce. Changes --- -Included a sample text for the JUnit tests to run (about 5MB; it is the complete works of Shakespeare). -Fixed some stream closings so they are enclosed within a try/catch with a finally for the closure. -Moved files from source to test/mapred. Summary --- Looking to add 3 new examples -- they can be added via the diff attached; some changes to the ExamplesDriver.java might be required however these files do work alone as well. I will also be attaching JUnit tests for these examples. I will post another review request for those. Diffs (updated) - http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/test/mapred/org/apache/hadoop/examples/TestWordStandardDeviation.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/test/mapred/org/apache/hadoop/examples/input/shaks12.txt PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/WordMedian.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/WordStandardDeviation.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/test/mapred/org/apache/hadoop/examples/TestWordMean.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/test/mapred/org/apache/hadoop/examples/TestWordMedian.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/WordMean.java PRE-CREATION Diff: https://reviews.apache.org/r/1091/diff Testing --- JUnit tests added -- reduced to 3 tests that all read from an input folder. Thanks, Plamen
Re: Review Request: Addition of 3 new examples -- WordMean, WordMedian, and WordStandardDeviation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1091/ --- (Updated 2011-07-13 18:27:11.020192) Review request for hadoop-mapreduce. Summary --- Looking to add 3 new examples -- they can be added via the diff attached; some changes to the ExamplesDriver.java might be required however these files do work alone as well. I will also be attaching JUnit tests for these examples. I will post another review request for those. Diffs - http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/test/WordStandardDeviationRobustTest.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/test/WordMeanRobustTest.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/test/WordMedianRobustTest.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/WordStandardDeviation.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/WordMedian.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/WordMean.java PRE-CREATION Diff: https://reviews.apache.org/r/1091/diff Testing (updated) --- JUnit tests added -- reduced to 3 tests that all read from an input folder. Thanks, Plamen
Review Request: Addition of 3 new examples -- WordMean, WordMedian, and WordStandardDeviation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1091/ --- Review request for hadoop-mapreduce. Summary --- Looking to add 3 new examples -- they can be added via the diff attached; some changes to the ExamplesDriver.java might be required however these files do work alone as well. I will also be attaching JUnit tests for these examples. I will post another review request for those. Diffs - http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/WordMean.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/WordMedian.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/WordStandardDeviation.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/test/WordMeanEmptyTest.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/test/WordMeanRobustTest.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/test/WordMeanTest.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/test/WordMedianEmptyTest.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/test/WordMedianTest.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/test/WordStandardDeviationEmptyTest.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/test/WordStandardDeviationTest.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/test/input/LICENSE.txt PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/test/input/shaks12.txt PRE-CREATION http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/test/robust_output/._SUCCESS.crc UNKNOWN http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/test/robust_output/.part-r-0.crc UNKNOWN http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce/src/examples/org/apache/hadoop/examples/test/robust_output/part-r-0 PRE-CREATION Diff: https://reviews.apache.org/r/1091/diff Testing --- JUnit tests will be added shortly. Thanks, Plamen
[jira] [Created] (MAPREDUCE-2669) Some new examples and test cases for them.
Some new examples and test cases for them. -- Key: MAPREDUCE-2669 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2669 Project: Hadoop Map/Reduce Issue Type: Test Components: examples Affects Versions: 0.22.0 Reporter: Plamen Jeliazkov Priority: Minor Looking to add some more examples such as Mean, Median, and Standard Deviation to the examples. I have some generic JUnit testcases as well, though I feel that they can be improved. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
