Ok so should I try and recompile and change the guava version to 11.0.2 in the pom ?
Kévin Moulart 2014-03-06 16:26 GMT+01:00 Sean Owen <sro...@gmail.com>: > That's gonna be a Guava version problem. I have seen variants of this > for a while. Hadoop still uses 11.0.2 even in HEAD and you can often > get away with using a later version in a project like this, even > though code that executes on Hadoop will use an older Guava than you > compiled against. This is an example of that gotcha. I think it may be > necessary to force Mahout to use 11.0.2 and change this code. > > I am having deja vu like this has come up before too. > > > > > > On Thu, Mar 6, 2014 at 3:23 PM, Kevin Moulart <kevinmoul...@gmail.com> > wrote: > > Hi thanks very much it seems to have worked ! > > Compiling with "mvn clean package -Dhadoop2.version=2.0.0-cdh4.6.0" works > > and I no longer have the error, but then when running tests that used to > > work with previous install like trainAdaptativeLogistic and then > > ValidateAdaptativeLogistic, the first works but the second yields an > error : > > > > bin/mahout validateAdaptiveLogistic --input > > /mnt/hdfs/user/myCompany/Echant/echant300k_wh.csv --model > > /mnt/hdfs/user/myCompany/Echant/Models/echnat.model --auc --scores > > --confusion. > > 14/03/06 15:53:42 WARN driver.MahoutDriver: No > > validateAdaptiveLogistic.props found on classpath, will use command-line > > arguments only > > Exception in thread "main" java.lang.NoSuchMethodError: > > com.google.common.collect.Queues.newArrayDeque()Ljava/util/ArrayDeque; > > at org.apache.mahout.math.stats.GroupTree$1.<init>(GroupTree.java:171) > > at org.apache.mahout.math.stats.GroupTree.iterator(GroupTree.java:169) > > at org.apache.mahout.math.stats.GroupTree.access$300(GroupTree.java:14) > > at org.apache.mahout.math.stats.GroupTree$2.iterator(GroupTree.java:317) > > at org.apache.mahout.math.stats.TDigest.add(TDigest.java:105) > > at org.apache.mahout.math.stats.TDigest.add(TDigest.java:88) > > at org.apache.mahout.math.stats.TDigest.add(TDigest.java:76) > > at > > > org.apache.mahout.math.stats.OnlineSummarizer.add(OnlineSummarizer.java:57) > > at > > > org.apache.mahout.classifier.sgd.ValidateAdaptiveLogistic.mainToOutput(ValidateAdaptiveLogistic.java:107) > > at > > > org.apache.mahout.classifier.sgd.ValidateAdaptiveLogistic.main(ValidateAdaptiveLogistic.java:63) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:606) > > at > > > org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) > > at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144) > > at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:195) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:606) > > at org.apache.hadoop.util.RunJar.main(RunJar.java:208) > > > > I'll try some other tests to see what's working and what's not. > > > > > > > > 2014-03-06 15:58 GMT+01:00 Gokhan Capan <gkhn...@gmail.com>: > > > >> Kevin, > >> > >> > >> From trunk, can you build mahout for hadoop2 using this command: > >> > >> mvn clean package -DskipTests=true > -Dhadoop2.version=<YOUR_HADOOP2_VERSION> > >> > >> > >> Then can you verify that you have the right hadoop jars with the > following > >> command: > >> > >> find . -name hadoop*.jar > >> > >> > >> > >> Gokhan > >> > >> > >> On Thu, Mar 6, 2014 at 3:26 PM, Kevin Moulart <kevinmoul...@gmail.com > >> >wrote: > >> > >> > Hi again, and thanks for the enthousiasm ! > >> > > >> > I did compile the trunk with the hadoop2 profile and, althoug it > didn't > >> > work at first because of some Canopy tests not passing, when I skipped > >> the > >> > tests it compiled and when I tested it afterward it passed. > >> > I used the version I have isntalled, so I just added the line : > >> > <hadoop2.version>2.0.0-cdh4.6.0</hadoop2.version> > >> > To the pom.xml and type : > >> > mvn -DskipTests clean install -Phadoop2 > >> > Then : > >> > mvn test > >> > > >> > Then I tried it with these settings : > >> > export HADOOP_HOME=/opt/cloudera/parcels/CDH/lib/hadoop > >> > export > >> > > >> > > >> > HADOOP_CLASSPATH=/home/myCompany/Downloads/mahout9/examples/target/mahout-examples-1.0-SNAPSHOT-job.jar > >> > export MAHOUT_HOME=/home/myCompany/Downloads/mahout9 > >> > > >> > And the command gives me this : > >> > [root@node01 mahout9]# bin/mahout > >> > MAHOUT_LOCAL is not set; adding HADOOP_CONF_DIR to classpath. > >> > Running on hadoop, using > /opt/cloudera/parcels/CDH/lib/hadoop/bin/hadoop > >> > and HADOOP_CONF_DIR=/etc/hadoop/conf > >> > MAHOUT-JOB: > >> > > >> > > >> > /home/myCompany/Downloads/mahout9/examples/target/mahout-examples-1.0-SNAPSHOT-job.jar > >> > Exception in thread "main" java.lang.NoSuchMethodError: > >> > org.apache.hadoop.util.ProgramDriver.driver([Ljava/lang/String;)V > >> > at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:122) > >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >> > at > >> > > >> > > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > >> > at > >> > > >> > > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> > at java.lang.reflect.Method.invoke(Method.java:606) > >> > at org.apache.hadoop.util.RunJar.main(RunJar.java:208) > >> > > >> > I even tried with : > >> > export HADOOP_HOME=/.../hadoop, > >> > export HADOOP_HOME=/.../hadoop-0.20-mapreduce > >> > export HADOOP_HOME=/.../hadoop-mapreduce > >> > > >> > And it still gives me the same result. > >> > > >> > And recompiling with <hadoop2.version>2.0.0</hadoop2.version> or > >> > <hadoop2.version>2.0.0-mr1-cdh4.6.0</hadoop2.version> didn't work. > >> > > >> > Any idea ? > >> > > >> > > >> > > >> > 2014-03-05 22:42 GMT+01:00 Andrew Musselman < > andrew.mussel...@gmail.com > >> >: > >> > > >> > > I mean "balance the risk aversion against the value of new features" > >> duh. > >> > > > >> > > > >> > > On Wed, Mar 5, 2014 at 1:39 PM, Andrew Musselman < > >> > > andrew.mussel...@gmail.com > >> > > > wrote: > >> > > > >> > > > Yeah, for sure; balancing clients' risk aversion to technical > >> features > >> > is > >> > > > why we often recommend vendor solutions. > >> > > > > >> > > > Having a little button to choose a newer version of a component in > >> the > >> > > > Manager UI (even with a confirmation dialog that said "Are you > sure? > >> > Are > >> > > > you crazy?") would be more palatable to some teams than installing > >> > > > tarballs, is what I'm getting at. > >> > > > > >> > > > > >> > > > On Wed, Mar 5, 2014 at 1:30 PM, Sean Owen <sro...@gmail.com> > wrote: > >> > > > > >> > > >> You can always install whatever version of anything on your > cluster > >> > > >> that you want. It may or may not work, but often happens to, at > >> least > >> > > >> for whatever you need it to do. > >> > > >> > >> > > >> It's just the same as it is without a packaged distribution -- > dump > >> > > >> new tarballs and cross your fingers. Nothing is weird or > different > >> > > >> about the setup or layout. That is the "here be dragons" > solution, > >> > > >> already > >> > > >> > >> > > >> You go with support from a packaged distribution when you want a > >> "here > >> > > >> be no dragons" solution. Everything else is by definition already > >> > > >> something you can and should do yourself outside of a packaged > >> > > >> distribution. And really -- you freely can, and it's not hard, if > >> you > >> > > >> know what you are doing. > >> > > >> > >> > > >> On Wed, Mar 5, 2014 at 9:15 PM, Andrew Musselman > >> > > >> <andrew.mussel...@gmail.com> wrote: > >> > > >> > Feels like just yesterday :) > >> > > >> > > >> > > >> > Consider this a feature request to have more flexible component > >> > > >> versioning, > >> > > >> > even with a caveat/"here be dragons" warning. I know that > >> > complicates > >> > > >> > things but people do use your releases a long time. I > personally > >> > > >> wished I > >> > > >> > could upgrade Pig on CDH 4 for new features but there was no > >> simple > >> > > way > >> > > >> on > >> > > >> > a managed cluster. > >> > > >> > > >> > > >> > > >> > > >> > On Wed, Mar 5, 2014 at 12:12 PM, Sean Owen <sro...@gmail.com> > >> > wrote: > >> > > >> > > >> > > >> >> I don't understand this -- CDH always bundles the latest > release. > >> > > >> >> > >> > > >> >> You know that CDH4 was released in July 2012, right? So it > >> included > >> > > >> >> 0.7 + patches. CDH5 includes 0.8 because 0.9 was released > about a > >> > > >> >> month after it began beta 2. > >> > > >> >> > >> > > >> >> CDH follows semantic versioning and won't introduce changes > that > >> > are > >> > > >> >> not backwards-compatible in a minor version update. 0.x > releases > >> of > >> > > >> >> Mahout act like major version changes -- not backwards > >> compatible. > >> > So > >> > > >> >> 4.x will always be 0.7 and 5.x will always be 0.8. > >> > > >> >> > >> > > >> >> On Wed, Mar 5, 2014 at 5:34 PM, Dmitriy Lyubimov < > >> > dlie...@gmail.com> > >> > > >> >> wrote: > >> > > >> >> > On Wed, Mar 5, 2014 at 9:08 AM, Sean Owen <sro...@gmail.com > > > >> > > wrote: > >> > > >> >> > > >> > > >> >> >> I don't follow what here makes you say they are "cut down" > >> > > releases? > >> > > >> >> >> > >> > > >> >> > > >> > > >> >> > meaning it seems to be pretty much 2 releases behind the > >> > official. > >> > > >> But i > >> > > >> >> > definitely don't follow CDH developments in this department, > >> you > >> > > >> seem in > >> > > >> >> a > >> > > >> >> > better position to explain the existing patchlevel there so > I > >> > defer > >> > > >> to > >> > > >> >> you > >> > > >> >> > to explain why this patchlevel is not there. > >> > > >> >> > > >> > > >> >> > -d > >> > > >> >> > >> > > >> > >> > > > > >> > > > > >> > > > >> > > >> > > >> > > >> > -- > >> > Kévin Moulart > >> > GSM France : +33 7 81 06 10 10 > >> > GSM Belgique : +32 473 85 23 85 > >> > Téléphone fixe : +32 2 771 88 45 > >> > > >> > > > > > > > > -- > > Kévin Moulart > > GSM France : +33 7 81 06 10 10 > > GSM Belgique : +32 473 85 23 85 > > Téléphone fixe : +32 2 771 88 45 >
