[ https://issues.apache.org/jira/browse/MAHOUT-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226548#comment-13226548 ]
jirapos...@reviews.apache.org commented on MAHOUT-822: ------------------------------------------------------ bq. On 2012-03-08 23:42:23, Dmitriy Lyubimov wrote: bq. > trunk/core/src/main/java/org/apache/mahout/common/HadoopUtil.java, line 247 bq. > <https://reviews.apache.org/r/4237/diff/1/?file=88913#file88913line247> bq. > bq. > is this really an issue with Hadoop 0.20.203? i don't think we care about classic hadoop 0.20.2 anymore. bq. bq. tom pierce wrote: bq. Hmm, I am not sure, but I think this is a comment we can omit. Without the new listStatus methods (but with the changes to other files), there is at least one test that won't compile. I'm willing to figure out who added this to the patch and ping them about it, but before I do, what should I ask them to clarify? Exactly which version of Hadoop they had in mind when making the changes to HadoopUtil? We've misinterpreted the comment - an empty array is returned to emulate 0.20.x behavior under 0.23.1. I applied the patch, reverted HadoopUtil and TestDistributedRowMatrix (which is the only class using these new methods), and that test still passes under 0.20.x but not under 0.23.1. From the stacktrace it seems fs.listStatus will now throws an exception rather than returning an empty array when there are no matches. I can see this comment causing the same confusion in the future; I will plan to remove the "retain compatibility" comment from HadoopUtil (it appears 2x) before committing. - tom ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4237/#review5754 ----------------------------------------------------------- On 2012-03-08 01:24:21, tom pierce wrote: bq. bq. ----------------------------------------------------------- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4237/ bq. ----------------------------------------------------------- bq. bq. (Updated 2012-03-08 01:24:21) bq. bq. bq. Review request for mahout and Dmitriy Lyubimov. bq. bq. bq. Summary bq. ------- bq. bq. This is the current patch for MAHOUT-822 (as posted by Bilung Lee). bq. bq. bq. This addresses bug MAHOUT-822. bq. https://issues.apache.org/jira/browse/MAHOUT-822 bq. bq. bq. Diffs bq. ----- bq. bq. trunk/core/pom.xml 1296318 bq. trunk/core/src/main/java/org/apache/mahout/common/HadoopUtil.java 1296318 bq. trunk/core/src/test/java/org/apache/mahout/classifier/df/mapreduce/partial/MockContext.java 1296318 bq. trunk/core/src/test/java/org/apache/mahout/classifier/df/mapreduce/partial/PartialSequentialBuilder.java 1296318 bq. trunk/core/src/test/java/org/apache/mahout/classifier/df/mapreduce/partial/Step1MapperTest.java 1296318 bq. trunk/core/src/test/java/org/apache/mahout/clustering/canopy/TestCanopyCreation.java 1296318 bq. trunk/core/src/test/java/org/apache/mahout/clustering/classify/ClusterClassificationDriverTest.java 1296318 bq. trunk/core/src/test/java/org/apache/mahout/clustering/kmeans/TestKmeansClustering.java 1296318 bq. trunk/core/src/test/java/org/apache/mahout/clustering/meanshift/TestMeanShift.java 1296318 bq. trunk/core/src/test/java/org/apache/mahout/common/DummyCounter.java 1296318 bq. trunk/core/src/test/java/org/apache/mahout/common/DummyRecordWriter.java 1296318 bq. trunk/core/src/test/java/org/apache/mahout/common/DummyStatusReporter.java 1296318 bq. trunk/core/src/test/java/org/apache/mahout/math/hadoop/TestDistributedRowMatrix.java 1296318 bq. trunk/integration/src/test/java/org/apache/mahout/clustering/TestClusterDumper.java 1296318 bq. trunk/pom.xml 1296318 bq. bq. Diff: https://reviews.apache.org/r/4237/diff bq. bq. bq. Testing bq. ------- bq. bq. Passes unit tests under default config as well as under hadoop 0.23.1. bq. bq. bq. Thanks, bq. bq. tom bq. bq. > Mahout needs to be made compatible with Hadoop .23 releases > ----------------------------------------------------------- > > Key: MAHOUT-822 > URL: https://issues.apache.org/jira/browse/MAHOUT-822 > Project: Mahout > Issue Type: Improvement > Components: build > Affects Versions: 0.6 > Reporter: Roman Shaposhnik > Labels: bigtop > Attachments: MAHOUT-822-build.patch.txt, MAHOUT-822.patch, > MAHOUT-822.patch, MAHOUT-822.patch, MAHOUT-822.patch, MAHOUT-822.patch, > MAHOUT-822.patch, MAHOUT-822.patch, MAHOUT-822.patch > > > As part of the Hadoop stack integration project (Apache Bigtop) we are now > trying to compile Mahout's upcoming 0.6 release against Hadoop 0.22 and 0.23. > I'm attaching the patch to Mahout's Maven build system that made it possible. > I would also like to request help in solving the real issues that poped > up when we tried to compile Mahout: > http://bigtop01.cloudera.org:8080/job/Bigtop-hadoop22/COMPONENT=mahout,label=centos5/6/console > {noformat} > [ERROR] > /mnt/jenkins/workspace/workspace/Bigtop-hadoop22/COMPONENT/mahout/label/centos5/build/mahout/rpm/BUILD/apache-mahout-c298f70/core/src/test/java/org/apache/mahout/df/mapreduce/partial/Step0JobTest.java:[182,33] > org.apache.hadoop.mapreduce.TaskAttemptContext is abstract; cannot be > instantiated > [ERROR] > /mnt/jenkins/workspace/workspace/Bigtop-hadoop22/COMPONENT/mahout/label/centos5/build/mahout/rpm/BUILD/apache-mahout-c298f70/core/src/test/java/org/apache/mahout/df/mapreduce/partial/Step0JobTest.java:[218,9] > org.apache.mahout.df.mapreduce.partial.Step0JobTest.Step0Context is not > abstract and does not override abstract method getInputSplit() in > org.apache.hadoop.mapreduce.MapContext > [ERROR] > /mnt/jenkins/workspace/workspace/Bigtop-hadoop22/COMPONENT/mahout/label/centos5/build/mahout/rpm/BUILD/apache-mahout-c298f70/core/src/test/java/org/apache/mahout/df/mapreduce/partial/Step0JobTest.java:[229,12] > cannot find symbol > [ERROR] symbol : constructor > Context(org.apache.hadoop.conf.Configuration,org.apache.hadoop.mapreduce.TaskAttemptID,<nulltype>,<nulltype>,<nulltype>,<nulltype>,<nulltype>) > [ERROR] location: class org.apache.hadoop.mapreduce.Mapper.Context > [ERROR] > /mnt/jenkins/workspace/workspace/Bigtop-hadoop22/COMPONENT/mahout/label/centos5/build/mahout/rpm/BUILD/apache-mahout-c298f70/core/src/test/java/org/apache/mahout/common/DummyRecordWriter.java:[68,18] > org.apache.hadoop.mapreduce.Mapper.Context is abstract; cannot be > instantiated > [ERROR] > /mnt/jenkins/workspace/workspace/Bigtop-hadoop22/COMPONENT/mahout/label/centos5/build/mahout/rpm/BUILD/apache-mahout-c298f70/core/src/test/java/org/apache/mahout/common/DummyRecordWriter.java:[77,19] > org.apache.hadoop.mapreduce.Reducer.Context is abstract; cannot be > instantiated > [ERROR] > /mnt/jenkins/workspace/workspace/Bigtop-hadoop22/COMPONENT/mahout/label/centos5/build/mahout/rpm/BUILD/apache-mahout-c298f70/core/src/test/java/org/apache/mahout/df/mapreduce/partial/PartialSequentialBuilder.java:[110,30] > org.apache.hadoop.mapreduce.TaskAttemptContext is abstract; cannot be > instantiated > [ERROR] > /mnt/jenkins/workspace/workspace/Bigtop-hadoop22/COMPONENT/mahout/label/centos5/build/mahout/rpm/BUILD/apache-mahout-c298f70/core/src/test/java/org/apache/mahout/df/mapreduce/partial/PartialSequentialBuilder.java:[206,28] > org.apache.hadoop.mapreduce.JobContext is abstract; cannot be instantiated > [ERROR] > /mnt/jenkins/workspace/workspace/Bigtop-hadoop22/COMPONENT/mahout/label/centos5/build/mahout/rpm/BUILD/apache-mahout-c298f70/core/src/test/java/org/apache/mahout/df/mapreduce/partial/PartialSequentialBuilder.java:[227,30] > org.apache.hadoop.mapreduce.TaskAttemptContext is abstract; cannot be > instantiated > [ERROR] > /mnt/jenkins/workspace/workspace/Bigtop-hadoop22/COMPONENT/mahout/label/centos5/build/mahout/rpm/BUILD/apache-mahout-c298f70/core/src/test/java/org/apache/mahout/df/mapreduce/partial/MockContext.java:[30,6] > org.apache.mahout.df.mapreduce.partial.MockContext is not abstract and does > not override abstract method getInputSplit() in > org.apache.hadoop.mapreduce.MapContext > [ERROR] > /mnt/jenkins/workspace/workspace/Bigtop-hadoop22/COMPONENT/mahout/label/centos5/build/mahout/rpm/BUILD/apache-mahout-c298f70/core/src/test/java/org/apache/mahout/df/mapreduce/partial/MockContext.java:[38,10] > cannot find symbol > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira