Indeed it causes compile errors : [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project mahout-math: Compilation failure [ERROR] /home/myCompny/Downloads/mahout9/math/src/main/java/org/apache/mahout/math/stats/GroupTree.java:[171,31] cannot find symbol [ERROR] symbol: method newArrayDeque() [ERROR] location: class com.google.common.collect.Queues So I'll dig in the code to find where to replace and find an equivalent.
Thanks for your help ! Kévin Moulart 2014-03-06 16:36 GMT+01:00 Sean Owen <sro...@gmail.com>: > If I'm right, then it will cause compile errors, but then, you just > fix those by replacing some Guava constructs with equivalent Java or > older Guava code. IIRC it is fairly trivial. > > And in fact probably should not use Guava 12+ methods for this reason > even if compiling against 12+. And in fact I thought someone cleaned > that up... > > On Thu, Mar 6, 2014 at 3:34 PM, Kevin Moulart <kevinmoul...@gmail.com> > wrote: > > Ok so should I try and recompile and change the guava version to 11.0.2 > in > > the pom ? > > > > Kévin Moulart > > > > > > 2014-03-06 16:26 GMT+01:00 Sean Owen <sro...@gmail.com>: > > > >> That's gonna be a Guava version problem. I have seen variants of this > >> for a while. Hadoop still uses 11.0.2 even in HEAD and you can often > >> get away with using a later version in a project like this, even > >> though code that executes on Hadoop will use an older Guava than you > >> compiled against. This is an example of that gotcha. I think it may be > >> necessary to force Mahout to use 11.0.2 and change this code. > >> > >> I am having deja vu like this has come up before too. > >> > >> > >> > >> > >> > >> On Thu, Mar 6, 2014 at 3:23 PM, Kevin Moulart <kevinmoul...@gmail.com> > >> wrote: > >> > Hi thanks very much it seems to have worked ! > >> > Compiling with "mvn clean package -Dhadoop2.version=2.0.0-cdh4.6.0" > works > >> > and I no longer have the error, but then when running tests that used > to > >> > work with previous install like trainAdaptativeLogistic and then > >> > ValidateAdaptativeLogistic, the first works but the second yields an > >> error : > >> > > >> > bin/mahout validateAdaptiveLogistic --input > >> > /mnt/hdfs/user/myCompany/Echant/echant300k_wh.csv --model > >> > /mnt/hdfs/user/myCompany/Echant/Models/echnat.model --auc --scores > >> > --confusion. > >> > 14/03/06 15:53:42 WARN driver.MahoutDriver: No > >> > validateAdaptiveLogistic.props found on classpath, will use > command-line > >> > arguments only > >> > Exception in thread "main" java.lang.NoSuchMethodError: > >> > com.google.common.collect.Queues.newArrayDeque()Ljava/util/ArrayDeque; > >> > at org.apache.mahout.math.stats.GroupTree$1.<init>(GroupTree.java:171) > >> > at > org.apache.mahout.math.stats.GroupTree.iterator(GroupTree.java:169) > >> > at > org.apache.mahout.math.stats.GroupTree.access$300(GroupTree.java:14) > >> > at > org.apache.mahout.math.stats.GroupTree$2.iterator(GroupTree.java:317) > >> > at org.apache.mahout.math.stats.TDigest.add(TDigest.java:105) > >> > at org.apache.mahout.math.stats.TDigest.add(TDigest.java:88) > >> > at org.apache.mahout.math.stats.TDigest.add(TDigest.java:76) > >> > at > >> > > >> > org.apache.mahout.math.stats.OnlineSummarizer.add(OnlineSummarizer.java:57) > >> > at > >> > > >> > org.apache.mahout.classifier.sgd.ValidateAdaptiveLogistic.mainToOutput(ValidateAdaptiveLogistic.java:107) > >> > at > >> > > >> > org.apache.mahout.classifier.sgd.ValidateAdaptiveLogistic.main(ValidateAdaptiveLogistic.java:63) > >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >> > at > >> > > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > >> > at > >> > > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> > at java.lang.reflect.Method.invoke(Method.java:606) > >> > at > >> > > >> > org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) > >> > at > org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144) > >> > at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:195) > >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >> > at > >> > > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > >> > at > >> > > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> > at java.lang.reflect.Method.invoke(Method.java:606) > >> > at org.apache.hadoop.util.RunJar.main(RunJar.java:208) > >> > > >> > I'll try some other tests to see what's working and what's not. > >> > > >> > > >> > > >> > 2014-03-06 15:58 GMT+01:00 Gokhan Capan <gkhn...@gmail.com>: > >> > > >> >> Kevin, > >> >> > >> >> > >> >> From trunk, can you build mahout for hadoop2 using this command: > >> >> > >> >> mvn clean package -DskipTests=true > >> -Dhadoop2.version=<YOUR_HADOOP2_VERSION> > >> >> > >> >> > >> >> Then can you verify that you have the right hadoop jars with the > >> following > >> >> command: > >> >> > >> >> find . -name hadoop*.jar > >> >> > >> >> > >> >> > >> >> Gokhan > >> >> > >> >> > >> >> On Thu, Mar 6, 2014 at 3:26 PM, Kevin Moulart < > kevinmoul...@gmail.com > >> >> >wrote: > >> >> > >> >> > Hi again, and thanks for the enthousiasm ! > >> >> > > >> >> > I did compile the trunk with the hadoop2 profile and, althoug it > >> didn't > >> >> > work at first because of some Canopy tests not passing, when I > skipped > >> >> the > >> >> > tests it compiled and when I tested it afterward it passed. > >> >> > I used the version I have isntalled, so I just added the line : > >> >> > <hadoop2.version>2.0.0-cdh4.6.0</hadoop2.version> > >> >> > To the pom.xml and type : > >> >> > mvn -DskipTests clean install -Phadoop2 > >> >> > Then : > >> >> > mvn test > >> >> > > >> >> > Then I tried it with these settings : > >> >> > export HADOOP_HOME=/opt/cloudera/parcels/CDH/lib/hadoop > >> >> > export > >> >> > > >> >> > > >> >> > >> > HADOOP_CLASSPATH=/home/myCompany/Downloads/mahout9/examples/target/mahout-examples-1.0-SNAPSHOT-job.jar > >> >> > export MAHOUT_HOME=/home/myCompany/Downloads/mahout9 > >> >> > > >> >> > And the command gives me this : > >> >> > [root@node01 mahout9]# bin/mahout > >> >> > MAHOUT_LOCAL is not set; adding HADOOP_CONF_DIR to classpath. > >> >> > Running on hadoop, using > >> /opt/cloudera/parcels/CDH/lib/hadoop/bin/hadoop > >> >> > and HADOOP_CONF_DIR=/etc/hadoop/conf > >> >> > MAHOUT-JOB: > >> >> > > >> >> > > >> >> > >> > /home/myCompany/Downloads/mahout9/examples/target/mahout-examples-1.0-SNAPSHOT-job.jar > >> >> > Exception in thread "main" java.lang.NoSuchMethodError: > >> >> > org.apache.hadoop.util.ProgramDriver.driver([Ljava/lang/String;)V > >> >> > at > org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:122) > >> >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >> >> > at > >> >> > > >> >> > > >> >> > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > >> >> > at > >> >> > > >> >> > > >> >> > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> >> > at java.lang.reflect.Method.invoke(Method.java:606) > >> >> > at org.apache.hadoop.util.RunJar.main(RunJar.java:208) > >> >> > > >> >> > I even tried with : > >> >> > export HADOOP_HOME=/.../hadoop, > >> >> > export HADOOP_HOME=/.../hadoop-0.20-mapreduce > >> >> > export HADOOP_HOME=/.../hadoop-mapreduce > >> >> > > >> >> > And it still gives me the same result. > >> >> > > >> >> > And recompiling with <hadoop2.version>2.0.0</hadoop2.version> or > >> >> > <hadoop2.version>2.0.0-mr1-cdh4.6.0</hadoop2.version> didn't work. > >> >> > > >> >> > Any idea ? > >> >> > > >> >> > > >> >> > > >> >> > 2014-03-05 22:42 GMT+01:00 Andrew Musselman < > >> andrew.mussel...@gmail.com > >> >> >: > >> >> > > >> >> > > I mean "balance the risk aversion against the value of new > features" > >> >> duh. > >> >> > > > >> >> > > > >> >> > > On Wed, Mar 5, 2014 at 1:39 PM, Andrew Musselman < > >> >> > > andrew.mussel...@gmail.com > >> >> > > > wrote: > >> >> > > > >> >> > > > Yeah, for sure; balancing clients' risk aversion to technical > >> >> features > >> >> > is > >> >> > > > why we often recommend vendor solutions. > >> >> > > > > >> >> > > > Having a little button to choose a newer version of a > component in > >> >> the > >> >> > > > Manager UI (even with a confirmation dialog that said "Are you > >> sure? > >> >> > Are > >> >> > > > you crazy?") would be more palatable to some teams than > installing > >> >> > > > tarballs, is what I'm getting at. > >> >> > > > > >> >> > > > > >> >> > > > On Wed, Mar 5, 2014 at 1:30 PM, Sean Owen <sro...@gmail.com> > >> wrote: > >> >> > > > > >> >> > > >> You can always install whatever version of anything on your > >> cluster > >> >> > > >> that you want. It may or may not work, but often happens to, > at > >> >> least > >> >> > > >> for whatever you need it to do. > >> >> > > >> > >> >> > > >> It's just the same as it is without a packaged distribution -- > >> dump > >> >> > > >> new tarballs and cross your fingers. Nothing is weird or > >> different > >> >> > > >> about the setup or layout. That is the "here be dragons" > >> solution, > >> >> > > >> already > >> >> > > >> > >> >> > > >> You go with support from a packaged distribution when you > want a > >> >> "here > >> >> > > >> be no dragons" solution. Everything else is by definition > already > >> >> > > >> something you can and should do yourself outside of a packaged > >> >> > > >> distribution. And really -- you freely can, and it's not > hard, if > >> >> you > >> >> > > >> know what you are doing. > >> >> > > >> > >> >> > > >> On Wed, Mar 5, 2014 at 9:15 PM, Andrew Musselman > >> >> > > >> <andrew.mussel...@gmail.com> wrote: > >> >> > > >> > Feels like just yesterday :) > >> >> > > >> > > >> >> > > >> > Consider this a feature request to have more flexible > component > >> >> > > >> versioning, > >> >> > > >> > even with a caveat/"here be dragons" warning. I know that > >> >> > complicates > >> >> > > >> > things but people do use your releases a long time. I > >> personally > >> >> > > >> wished I > >> >> > > >> > could upgrade Pig on CDH 4 for new features but there was no > >> >> simple > >> >> > > way > >> >> > > >> on > >> >> > > >> > a managed cluster. > >> >> > > >> > > >> >> > > >> > > >> >> > > >> > On Wed, Mar 5, 2014 at 12:12 PM, Sean Owen < > sro...@gmail.com> > >> >> > wrote: > >> >> > > >> > > >> >> > > >> >> I don't understand this -- CDH always bundles the latest > >> release. > >> >> > > >> >> > >> >> > > >> >> You know that CDH4 was released in July 2012, right? So it > >> >> included > >> >> > > >> >> 0.7 + patches. CDH5 includes 0.8 because 0.9 was released > >> about a > >> >> > > >> >> month after it began beta 2. > >> >> > > >> >> > >> >> > > >> >> CDH follows semantic versioning and won't introduce changes > >> that > >> >> > are > >> >> > > >> >> not backwards-compatible in a minor version update. 0.x > >> releases > >> >> of > >> >> > > >> >> Mahout act like major version changes -- not backwards > >> >> compatible. > >> >> > So > >> >> > > >> >> 4.x will always be 0.7 and 5.x will always be 0.8. > >> >> > > >> >> > >> >> > > >> >> On Wed, Mar 5, 2014 at 5:34 PM, Dmitriy Lyubimov < > >> >> > dlie...@gmail.com> > >> >> > > >> >> wrote: > >> >> > > >> >> > On Wed, Mar 5, 2014 at 9:08 AM, Sean Owen < > sro...@gmail.com > >> > > >> >> > > wrote: > >> >> > > >> >> > > >> >> > > >> >> >> I don't follow what here makes you say they are "cut > down" > >> >> > > releases? > >> >> > > >> >> >> > >> >> > > >> >> > > >> >> > > >> >> > meaning it seems to be pretty much 2 releases behind the > >> >> > official. > >> >> > > >> But i > >> >> > > >> >> > definitely don't follow CDH developments in this > department, > >> >> you > >> >> > > >> seem in > >> >> > > >> >> a > >> >> > > >> >> > better position to explain the existing patchlevel there > so > >> I > >> >> > defer > >> >> > > >> to > >> >> > > >> >> you > >> >> > > >> >> > to explain why this patchlevel is not there. > >> >> > > >> >> > > >> >> > > >> >> > -d > >> >> > > >> >> > >> >> > > >> > >> >> > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > > >> >> > > >> >> > -- > >> >> > Kévin Moulart > >> >> > GSM France : +33 7 81 06 10 10 > >> >> > GSM Belgique : +32 473 85 23 85 > >> >> > Téléphone fixe : +32 2 771 88 45 > >> >> > > >> >> > >> > > >> > > >> > > >> > -- > >> > Kévin Moulart > >> > GSM France : +33 7 81 06 10 10 > >> > GSM Belgique : +32 473 85 23 85 > >> > Téléphone fixe : +32 2 771 88 45 > >> >
