Thanks Andrei. That fixed all the issues and I am able to launch a hadoop core cluster. I do wish maven would do more than nearest first resolution ( at least some warnings ) and not relegate ensuing issues to the user to figure out.
Arun On Sat, Oct 8, 2011 at 2:16 AM, Andrei Savu <[email protected]> wrote: > I think that if you are upgrading jclouds you also need to upgrade the > guava version in Whirr (root pom.xml) to match the one used by jclouds. > > -- Andrei Savu > > On Sat, Oct 8, 2011 at 1:25 AM, Arun Ramakrishnan < > [email protected]> wrote: > >> yup, I did a "mvn clean install assembly:assembly" on a fresh checkout >> with it pointing to a current build of jclouds 1.2.0-SNAPSHOT and >> jsr305-1.3.9 added as a dependency to whirr-core. >> And I get the same errors while using the command line. >> >> thanks >> Arun >> >> >> On Fri, Oct 7, 2011 at 5:29 AM, Andrei Savu <[email protected]>wrote: >> >>> Looks like a guava classpath ordering issue. Are you seeing this when >>> building a clean checkout? >>> On Oct 7, 2011 2:55 AM, "Arun Ramakrishnan" <[email protected]> >>> wrote: >>> >>>> Alas! While trying to run from the command line , I run into the >>>> following issue. Any ideas ? >>>> >>>> ######################## >>>> whirr-0.7.0-SNAPSHOT>bin/whirr launch-cluster --config >>>> ../whirr_cdh.properties >>>> Exception in thread "main" >>>> com.google.common.collect.ComputationException: >>>> java.lang.NoClassDefFoundError: com/google/common/cache/CacheLoader >>>> at >>>> com.google.common.collect.ComputingConcurrentHashMap$ComputingValueReference.compute(ComputingConcurrentHashMap.java:325) >>>> at >>>> com.google.common.collect.ComputingConcurrentHashMap$ComputingSegment.compute(ComputingConcurrentHashMap.java:140) >>>> at >>>> com.google.common.collect.ComputingConcurrentHashMap.apply(ComputingConcurrentHashMap.java:71) >>>> at >>>> com.google.common.collect.MapMaker$ComputingMapAdapter.get(MapMaker.java:848) >>>> at org.apache.whirr.service.ComputeCache.apply(ComputeCache.java:66) >>>> at org.apache.whirr.service.ComputeCache.apply(ComputeCache.java:57) >>>> at >>>> org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:62) >>>> at >>>> org.apache.whirr.ClusterController.launchCluster(ClusterController.java:106) >>>> at >>>> org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:62) >>>> at org.apache.whirr.cli.Main.run(Main.java:64) >>>> at org.apache.whirr.cli.Main.main(Main.java:97) >>>> Caused by: java.lang.NoClassDefFoundError: >>>> com/google/common/cache/CacheLoader >>>> at >>>> org.jclouds.aws.ec2.AWSEC2ContextBuilder.addClientModule(AWSEC2ContextBuilder.java:42) >>>> at >>>> org.jclouds.rest.RestContextBuilder.addClientModuleIfNotPresent(RestContextBuilder.java:267) >>>> at >>>> org.jclouds.rest.RestContextBuilder.buildInjector(RestContextBuilder.java:179) >>>> at >>>> org.jclouds.compute.ComputeServiceContextBuilder.buildInjector(ComputeServiceContextBuilder.java:52) >>>> at >>>> org.jclouds.compute.ComputeServiceContextBuilder.buildComputeServiceContext(ComputeServiceContextBuilder.java:65) >>>> at >>>> org.jclouds.compute.ComputeServiceContextFactory.buildContextUnwrappingExceptions(ComputeServiceContextFactory.java:70) >>>> at >>>> org.jclouds.compute.ComputeServiceContextFactory.createContext(ComputeServiceContextFactory.java:121) >>>> at >>>> org.apache.whirr.service.ComputeCache$1.apply(ComputeCache.java:82) >>>> at >>>> org.apache.whirr.service.ComputeCache$1.apply(ComputeCache.java:72) >>>> at >>>> com.google.common.collect.ComputingConcurrentHashMap$ComputingValueReference.compute(ComputingConcurrentHashMap.java:316) >>>> ... 10 more >>>> Caused by: java.lang.ClassNotFoundException: >>>> com.google.common.cache.CacheLoader >>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202) >>>> at java.security.AccessController.doPrivileged(Native Method) >>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190) >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306) >>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247) >>>> ... 20 more >>>> ########################################################### >>>> >>>> On Thu, Oct 6, 2011 at 3:51 PM, Arun Ramakrishnan < >>>> [email protected]> wrote: >>>> >>>>> Hurray! Then I included jsr305 in whirr-core and everything builds >>>>> fine. >>>>> >>>>> While following your suggestion, I deleted my ~/.m2 directory and did a >>>>> mvn clean install. Is that what you meant by clean the cache ? . I still >>>>> had >>>>> the same problem. >>>>> >>>>> here is some info about my environment. >>>>> ####################################### >>>>> /Users/arun/workspace/whirr-trunk>mvn --version >>>>> Apache Maven 3.0.3 (r1075438; 2011-02-28 09:31:09-0800) >>>>> Maven home: /usr/share/maven >>>>> Java version: 1.6.0_26, vendor: Apple Inc. >>>>> Java home: >>>>> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home >>>>> Default locale: en_US, platform encoding: MacRoman >>>>> OS name: "mac os x", version: "10.7.1", arch: "x86_64", family: "mac" >>>>> ####################################### >>>>> >>>>> >>>>> thanks >>>>> Arun >>>>> >>>>> >>>>> >>>>> On Thu, Oct 6, 2011 at 1:41 PM, Andrei Savu <[email protected]>wrote: >>>>> >>>>>> It seems like this is somehow related to Guava. >>>>>> >>>>>> http://stackoverflow.com/questions/3800033/guava-r07-gwt-and-javax-annotation-nullable >>>>>> >>>>>> Can you retry to build Whirr after removing the maven cache? I'm not >>>>>> sure this is the problem you are seeing. >>>>>> >>>>>> -- Andrei Savu >>>>>> >>>>>> On Thu, Oct 6, 2011 at 11:40 PM, Andrei Savu >>>>>> <[email protected]>wrote: >>>>>> >>>>>>> I think I've seen this in the past on a different project. Can you >>>>>>> tell me more about os, IDE, jdk version and maven version? >>>>>>> >>>>>>> -- Andrei Savu >>>>>>> >>>>>>> On Thu, Oct 6, 2011 at 10:21 PM, Arun Ramakrishnan < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Thanks. >>>>>>>> >>>>>>>> I applied the patch to jclouds trunk and I ran into this problem. I >>>>>>>> didn't spend too much time investigating it. But, let me know if you >>>>>>>> an idea >>>>>>>> whats wrong. >>>>>>>> >>>>>>>> ############################################################# >>>>>>>> [INFO] Apache Whirr Core ................................. FAILURE >>>>>>>> [7.074s] >>>>>>>> >>>>>>>> ... >>>>>>>> >>>>>>>> [ERROR] Failed to execute goal >>>>>>>> org.apache.maven.plugins:maven-compiler-plugin:2.0.2:compile >>>>>>>> (default-compile) on project whirr-core: Compilation failure: >>>>>>>> Compilation >>>>>>>> failure: >>>>>>>> [ERROR] >>>>>>>> /Users/arun/workspace/whirr-trunk/core/src/main/java/org/apache/whirr/RolePredicates.java:[29,23] >>>>>>>> cannot find symbol >>>>>>>> [ERROR] symbol : class Nullable >>>>>>>> [ERROR] location: package javax.annotation >>>>>>>> [ERROR] >>>>>>>> [ERROR] >>>>>>>> /Users/arun/workspace/whirr-trunk/core/src/main/java/org/apache/whirr/RolePredicates.java:[108,28] >>>>>>>> cannot find symbol >>>>>>>> [ERROR] symbol: class Nullable >>>>>>>> [ERROR] public boolean apply(@Nullable Instance input) { >>>>>>>> [ERROR] -> [Help 1] >>>>>>>> ########################################################### >>>>>>>> >>>>>>>> thanks >>>>>>>> Arun >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Oct 4, 2011 at 10:45 PM, Andrei Savu <[email protected] >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> mvn package assembly:assembly will generate the release tarball >>>>>>>>> from the source code. >>>>>>>>> >>>>>>>>> -- Andrei >>>>>>>>> >>>>>>>>> On Wed, Oct 5, 2011 at 4:07 AM, Arun Ramakrishnan < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Its always been less than 10 security groups in total. I will give >>>>>>>>>> this a try however. >>>>>>>>>> I am able to "mvn install" successfully in whirr-trunk. Is there a >>>>>>>>>> maven goal or a script that does the final packaging for command >>>>>>>>>> line use ? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Oct 4, 2011 at 2:08 PM, Tom White <[email protected]>wrote: >>>>>>>>>> >>>>>>>>>>> Hi Arun, >>>>>>>>>>> >>>>>>>>>>> I had this issue too - I think it is a consequence of using an >>>>>>>>>>> account >>>>>>>>>>> with a large number of existing groups. There's a fix in >>>>>>>>>>> https://issues.apache.org/jira/browse/WHIRR-381 that you might >>>>>>>>>>> like to >>>>>>>>>>> try - this will be fixed in the next release. >>>>>>>>>>> >>>>>>>>>>> Cheers, >>>>>>>>>>> Tom >>>>>>>>>>> >>>>>>>>>>> On Tue, Oct 4, 2011 at 1:30 PM, Arun Ramakrishnan >>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>> > I am having this issue upgrading to 0.6. >>>>>>>>>>> > >>>>>>>>>>> > During the security group creation, the internal access rules >>>>>>>>>>> within the >>>>>>>>>>> > cluster don't get added, consequentially blocking the datanodes >>>>>>>>>>> from being >>>>>>>>>>> > able to connect to the namenode. I added these rules manually >>>>>>>>>>> and it seems >>>>>>>>>>> > fine. >>>>>>>>>>> > >>>>>>>>>>> > This bug recurs for me consistently in my environment. >>>>>>>>>>> > >>>>>>>>>>> > log >>>>>>>>>>> > ######################################################## >>>>>>>>>>> > >>>>>>>>>>> > 2011-10-04 12:23:20,949 DEBUG [jclouds.compute] >>>>>>>>>>> (pool-3-thread-4) >> >>>>>>>>>>> > creating keyPair region(us-east-1) group(ccore27) >>>>>>>>>>> > 2011-10-04 12:23:20,949 DEBUG [jclouds.compute] >>>>>>>>>>> (pool-3-thread-2) >> >>>>>>>>>>> > creating keyPair region(us-east-1) group(ccore27) >>>>>>>>>>> > 2011-10-04 12:23:21,414 DEBUG [jclouds.compute] >>>>>>>>>>> (pool-3-thread-4) << created >>>>>>>>>>> > keyPair(jclouds#ccore27#us-east-1#72) >>>>>>>>>>> > 2011-10-04 12:23:21,414 DEBUG [jclouds.compute] >>>>>>>>>>> (pool-3-thread-4) >> >>>>>>>>>>> > creating securityGroup region(us-east-1) >>>>>>>>>>> name(jclouds#ccore27#us-east-1) >>>>>>>>>>> > 2011-10-04 12:23:21,692 DEBUG [jclouds.compute] >>>>>>>>>>> (pool-3-thread-4) << created >>>>>>>>>>> > securityGroup(jclouds#ccore27#us-east-1) >>>>>>>>>>> > 2011-10-04 12:23:21,692 DEBUG [jclouds.compute] >>>>>>>>>>> (pool-3-thread-4) >> >>>>>>>>>>> > authorizing securityGroup region(us-east-1) >>>>>>>>>>> name(jclouds#ccore27#us-east-1) >>>>>>>>>>> > port(22) >>>>>>>>>>> > 2011-10-04 12:23:21,926 DEBUG [jclouds.compute] >>>>>>>>>>> (pool-3-thread-4) << >>>>>>>>>>> > authorized securityGroup(jclouds#ccore27#us-east-1) >>>>>>>>>>> > 2011-10-04 12:23:21,926 DEBUG [jclouds.compute] >>>>>>>>>>> (pool-3-thread-4) >> >>>>>>>>>>> > authorizing securityGroup region(us-east-1) >>>>>>>>>>> name(jclouds#ccore27#us-east-1) >>>>>>>>>>> > permission to itself >>>>>>>>>>> > 2011-10-04 12:23:22,306 ERROR >>>>>>>>>>> > [org.apache.whirr.actions.BootstrapClusterAction] >>>>>>>>>>> (pool-3-thread-3) >>>>>>>>>>> > Unexpected error while starting 4 nodes, minimum 4 nodes for >>>>>>>>>>> > [hadoop-datanode, hadoop-tasktracker] of cluster ccore27 >>>>>>>>>>> > java.util.concurrent.ExecutionException: >>>>>>>>>>> java.lang.RuntimeException: >>>>>>>>>>> > request: POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1; >>>>>>>>>>> cause: >>>>>>>>>>> > java.lang.NullPointerException >>>>>>>>>>> > at >>>>>>>>>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) >>>>>>>>>>> > at java.util.concurrent.FutureTask.get(FutureTask.java:83) >>>>>>>>>>> > at >>>>>>>>>>> > >>>>>>>>>>> org.apache.whirr.actions.BootstrapClusterAction$StartupProcess.waitForOutcomes(BootstrapClusterAction.java:320) >>>>>>>>>>> > at >>>>>>>>>>> > >>>>>>>>>>> org.apache.whirr.actions.BootstrapClusterAction$StartupProcess.call(BootstrapClusterAction.java:273) >>>>>>>>>>> > at >>>>>>>>>>> > >>>>>>>>>>> org.apache.whirr.actions.BootstrapClusterAction$StartupProcess.call(BootstrapClusterAction.java:234) >>>>>>>>>>> > at >>>>>>>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >>>>>>>>>>> > at java.util.concurrent.FutureTask.run(FutureTask.java:138) >>>>>>>>>>> > at >>>>>>>>>>> > >>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>>>>>>>>> > at >>>>>>>>>>> > >>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>>>>>>>>> > at java.lang.Thread.run(Thread.java:680) >>>>>>>>>>> > Caused by: java.lang.RuntimeException: request: POST >>>>>>>>>>> > https://ec2.us-east-1.amazonaws.com/ HTTP/1.1; cause: >>>>>>>>>>> > java.lang.NullPointerException >>>>>>>>>>> > at >>>>>>>>>>> > >>>>>>>>>>> org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:152) >>>>>>>>>>> > at >>>>>>>>>>> org.jclouds.http.functions.ParseSax.parse(ParseSax.java:116) >>>>>>>>>>> > at >>>>>>>>>>> org.jclouds.http.functions.ParseSax.apply(ParseSax.java:78) >>>>>>>>>>> > at >>>>>>>>>>> org.jclouds.http.functions.ParseSax.apply(ParseSax.java:51) >>>>>>>>>>> > at >>>>>>>>>>> com.google.common.util.concurrent.Futures$4.apply(Futures.java:439) >>>>>>>>>>> > at >>>>>>>>>>> com.google.common.util.concurrent.Futures$4.apply(Futures.java:437) >>>>>>>>>>> > at >>>>>>>>>>> > >>>>>>>>>>> com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:713) >>>>>>>>>>> > at >>>>>>>>>>> > >>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>>>>>>>>> > at >>>>>>>>>>> > >>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>>>>>>>>> > at java.lang.Thread.run(Thread.java:680) >>>>>>>>>>> > at >>>>>>>>>>> > >>>>>>>>>>> org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:152) >>>>>>>>>>> > at >>>>>>>>>>> org.jclouds.http.functions.ParseSax.apply(ParseSax.java:80) >>>>>>>>>>> > at >>>>>>>>>>> org.jclouds.http.functions.ParseSax.apply(ParseSax.java:51) >>>>>>>>>>> > at >>>>>>>>>>> com.google.common.util.concurrent.Futures$4.apply(Futures.java:439) >>>>>>>>>>> > at >>>>>>>>>>> com.google.common.util.concurrent.Futures$4.apply(Futures.java:437) >>>>>>>>>>> > at >>>>>>>>>>> > >>>>>>>>>>> com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:713) >>>>>>>>>>> > ... 3 more >>>>>>>>>>> > >>>>>>>>>>> > ########################################################### >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > This are the rules that existed for the cluster created with >>>>>>>>>>> whirr 0.6 >>>>>>>>>>> > GROUP 673040621396 jclouds#ccore27#us-east-1 >>>>>>>>>>> > jclouds#ccore27#us-east-1 >>>>>>>>>>> > PERMISSION 673040621396 jclouds#ccore27#us-east-1 >>>>>>>>>>> ALLOWS tcp >>>>>>>>>>> > 22 22 FROM CIDR 0.0.0.0/0 >>>>>>>>>>> > PERMISSION 673040621396 jclouds#ccore27#us-east-1 >>>>>>>>>>> ALLOWS tcp >>>>>>>>>>> > 8020 8020 FROM CIDR 184.72.183.32/32 >>>>>>>>>>> > PERMISSION 673040621396 jclouds#ccore27#us-east-1 >>>>>>>>>>> ALLOWS tcp >>>>>>>>>>> > 8021 8021 FROM CIDR 184.72.183.32/32 >>>>>>>>>>> > PERMISSION 673040621396 jclouds#ccore27#us-east-1 >>>>>>>>>>> ALLOWS tcp >>>>>>>>>>> > 50030 50030 FROM CIDR 24.43.39.218/32 >>>>>>>>>>> > PERMISSION 673040621396 jclouds#ccore27#us-east-1 >>>>>>>>>>> ALLOWS tcp >>>>>>>>>>> > 50070 50070 FROM CIDR 24.43.39.218/32 >>>>>>>>>>> > >>>>>>>>>>> > I had to manually add >>>>>>>>>>> > PERMISSION 673040621396 jclouds#ccore27#us-east-1 >>>>>>>>>>> ALLOWS all >>>>>>>>>>> > FROM USER 673040621396 GRPNAME >>>>>>>>>>> jclouds#ccore27#us-east-1 >>>>>>>>>>> > to allow the datanodes to talk to the namenode. >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > Here is my config file with updated property names to match 0.6 >>>>>>>>>>> > ########################################## >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > whirr.cluster-name=ccore27 >>>>>>>>>>> > whirr.instance-templates=1 hadoop-jobtracker+hadoop-namenode,4 >>>>>>>>>>> > hadoop-datanode+hadoop-tasktracker >>>>>>>>>>> > >>>>>>>>>>> > whirr.provider=aws-ec2 >>>>>>>>>>> > whirr.identity=************************* >>>>>>>>>>> > whirr.credential=********************** >>>>>>>>>>> > whirr.private-key-file=/Users/arun/.ec2/hadoopkey >>>>>>>>>>> > whirr.public-key-file=/Users/arun/.ec2/hadoopkey.pub >>>>>>>>>>> > whirr.client-cidrs=24.43.39.218/32 >>>>>>>>>>> > >>>>>>>>>>> > whirr.location-id=us-east-1 >>>>>>>>>>> > whirr.hardware-id=c1.xlarge >>>>>>>>>>> > #c1.xlarge >>>>>>>>>>> > >>>>>>>>>>> > # Ubuntu 10.04 LTS Lucid. See http://alestic.com/ or >>>>>>>>>>> > http://aws.amazon.com/amis/4348 >>>>>>>>>>> > # ebs root only >>>>>>>>>>> > whirr.image-id=us-east-1/ami-4a0df923 >>>>>>>>>>> > >>>>>>>>>>> > whirr.hadoop.install-function=install_cdh_hadoop >>>>>>>>>>> > whirr.hadoop.configure-function=configure_cdh_hadoop >>>>>>>>>>> > >>>>>>>>>>> > ####################################### >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >> >
