Hi,
I would like to run a spark job on mesos on my machine locally. I
install mesos 0.22 and run mesos master and slave as a follows:
|./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos|
|./bin/mesos-slave.sh --master=127.0.0.1:5050|
I can open the http://127.0.0.1:5050/ and I can see mesos master.
I also started spark, and I can see the spark master on master:8080 with
worker.
I set enviroment variabels too:
MESOS_NATIVE_LIBRARY=/usr/local/lib/libmesos.dylib
SPARK_HOME=/Projects/Spark/spark
I tried to run spark on mesos as a follows:
valsConf =
SparkConf().setMaster("mesos://127.0.0.1:5050/").setAppName("MesosTest")
.set("spark.executor.uri","s3n://${AWS_AC_KEY}:${AWS_SEC_KEY}@com-collokia-test-ml/collokia/spark/run/spark-1.1.2-SNAPSHOT-bin-spark.tgz")
.set("spark.storage.memoryFraction","1")
.set("spark.core.connection.ack.wait.timeout","600")
.set("spark.akka.frameSize","500")
.set("spark.akka.timeout","100")
valsc = JavaSparkContext(sConf)
vallist = LinkedList<Int>()
for(iin0..1000){
list.add(i)
}
valrdd = sc.parallelize(list)
valresults = rdd.collect()
println(results)
But I got the following massages:
I1219 15:49:17.287739 621187072 sched.cpp:148] Version: 0.22.0
I1219 15:49:17.297101 624943104 sched.cpp:245] New master detected at
master@127.0.0.1:5050
I1219 15:49:17.300489 624943104 sched.cpp:255] No credentials provided.
Attempting to register without authentication
I1219 15:49:17.302489 624406528 sched.cpp:439] Framework registered with
20141219-110602-16777343-5050-658-0028
15:49:32.550 [Timer-0] WARN o.a.s.scheduler.TaskSchedulerImpl - Initial
job has not accepted any resources; check your cluster UI to ensure that
workers are registered and have sufficient memory
15:49:47.547 [Timer-0] WARN o.a.s.scheduler.TaskSchedulerImpl - Initial
job has not accepted any resources; check your cluster UI to ensure that
workers are registered and have sufficient memory
Do you have any idea, how can I fix it?
Thanks,
Istvan
