Hi all, Thanks for suggestions.
@Andre, Yes I'd checked all the binaries and also (spark/shark)-env.sh on all nodes. @Matei, I followed steps from https://github.com/amplab/shark/wiki/Running-Shark-on-a-Cluster So I had not built Spark / Shark on my machine, but was using pre-built binaries. I ran an sbt clean and sbt assembly for Spark (with due changes for CDH 4.4) for all nodes. Spark cluster loads up fine as before. For Shark,when I run : sudo SHARK_HADOOP_VERSION=2.0.0-cdh4.4.0 SHARK_YARN=true sbt/sbt assembly, It gives me an exception - [info] Compiling 96 Scala sources and 5 Java sources to /home/<>/shark-0.8.0-bin-cdh4/shark-0.8.0/target/scala-2.9.3/classes... [error] error while loading <root>, error in opening zip file scala.tools.nsc.MissingRequirementError: object scala not found. ... ... This is similar to the error which I saw when running shark-shell. I'm on Ubuntu 12.0.4 and have set JDK 7 as JAVA_HOME and also in PATH variable. Now I'm not even able to run ./shark (i.e. hive console) as my shark installation is corrupted :) I will re-sync the pre-built libraries for that. But still the original problem remains. Regards, Kapil Malik | [email protected]<mailto:[email protected]> From: Matei Zaharia [mailto:[email protected]] Sent: 13 November 2013 09:01 To: [email protected]<mailto:[email protected]> Subject: Re: shark-shell not launching in cluster It might mean one of your JARs is corrupted. Try doing sbt clean and then sbt assembly again. Matei On Nov 12, 2013, at 10:48 AM, Josh Rosen <[email protected]<mailto:[email protected]>> wrote: I've seen this "error: error while loading <root>, error in opening zip file" before, but I'm not exactly sure what causes it. Here's a JIRA discussing that error in earlier versions of Spark: https://spark-project.atlassian.net/browse/SPARK-692 On Tue, Nov 12, 2013 at 10:44 AM, Andre Schumacher <[email protected]<mailto:[email protected]>> wrote: Hi, Have you tried it in local mode? The error message seems to indicate problems with the classpath. So you may want to make sure that all the binaries are present on all nodes at the same location (in addition making sure the {spark,shark}-env.sh settings are correct on all nodes). Andre On 11/12/2013 02:57 AM, Kapil Malik wrote: > Hi all, > > I've a standalone spark + shark cluster, following steps on > https://github.com/amplab/shark/wiki/Running-Shark-on-a-Cluster > $SPARK_HOME/spark-shell works fine. I am able to access files, and perform > operations. > $SHARK_HOME/bin/shark also works fine, I am able to access hive tables and > perform operations. > > However, > $SHARK_HOME/bin/shark-shell does not work :( > > Logs - > Using Scala version 2.9.3 (OpenJDK 64-Bit Server VM, Java 1.6.0_27) > Initializing interpreter... > error: error while loading <root>, error in opening zip file > Failed to initialize compiler: object scala not found. > > I've set SCALA_HOME correctly in shark-env.sh and also in spark-env.sh. > > This failure restricts me to use sql2rdd operations. Can you please suggests > any steps to trouble shoot this OR make sql2rdd work with spark-shell (if not > shark-shell) > > Any suggestions ? > > Thanks and regards, > > Kapil Malik | > [email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>> > > >
