Spark stand alone cluster mode

Tue, 11 Mar 2014 06:05:26 -0700 

Hi,

I am new to spark.
I would like to run jobs  in Spark stand alone cluster mode.

No cluser managers other than spark is used. 
(https://spark.apache.org/docs/0.9.0/spark-standalone.html)
I have tried wordcount from spark shell and stand alone scala app.

The code reads input from HDFS and writes the results to HDFS. uses 2 worker 
nodes.

In spark-shell the wordcount is successful, how ever my effort to run stand 
alone programmes are in vain.

My environement
Ubuntu 12.04  - 32 bit
JAVA 1.7.0_51
I have installed spark @ $HOME/Downloads/spark-0.9.0-incubating
installed hadoop 2.2.0 as separate hduser and given permission to other users.
installed scala 2.10.3
installed sbt 0.13.1
Spark master act as HDFS master
I have one master and 2 worker nodes and HDFS is accessible in all nodes.
I downloaded example project and modified to use my spark cluster.
I started the sparkcluster at spark://192.168.0.138:7077
and hdfs://master:9000/
When I run the project as SPARK_HADOOP_VERSION=2.2.0 sbt run, I get following 
error

gino@master:~/Test/spark-example-project$ SPARK_HADOOP_VERSION=2.2.0 sbt run
[info] Loading project definition from 
/home/gino/Test/spark-example-project/project
[info] Set current project to spark-example-project (in build 
file:/home/gino/Test/spark-example-project/)
[info] Running com.Thinkpalm.spark.WordCountHDFS
[error] (run-main-0) java.lang.NoClassDefFoundError: 
org/apache/spark/SparkContext
java.lang.NoClassDefFoundError: org/apache/spark/SparkContext
    at com.Thinkpalm.spark.WordCountHDFS$.main(WordCountHDFS.scala:12)
    at com.Thinkpalm.spark.WordCountHDFS.main(WordCountHDFS.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.SparkContext
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at com.Thinkpalm.spark.WordCountHDFS$.main(WordCountHDFS.scala:12)
    at com.Thinkpalm.spark.WordCountHDFS.main(WordCountHDFS.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
[trace] Stack trace suppressed: run last compile:run for the full output.
java.lang.RuntimeException: Nonzero exit code: 1
    at scala.sys.package$.error(package.scala:27)
[trace] Stack trace suppressed: run last compile:run for the full output.
[error] (compile:run) Nonzero exit code: 1
[error] Total time: 0 s, completed Mar 11, 2014 2:54:54 PM

Could anyone give some pointers ... I have attached the project for reference.

Thanks and regards
Gino Mathews

Attachment: spark-example-project.tgz
Description: spark-example-project.tgz

Reply via email to