Hi, a couple-three things. First, is this a Gradle project? SBT? Regardless of the answer, convince yourself that you are getting this error from the command line before doing anything else. Eclipse is awesome and it's also really glitchy, I have seen too many times recently where something funky is happening in Eclipse but I can go to the shell and "gradle build" and "gradle run" things just fine.
Getting that out of the way, and I don't know yet how generally applicable this idea is, get rid of ALL hostnames and try with just IP adresses. I posted the results of some research I did this morning on SO: http://stackoverflow.com/questions/28453835/apache-sparck-error-could-not-connect-to-akka-tcp-sparkmaster/34499020#34499020 Note that what I focus on is getting all spurious config out of the way. Comment out all configs in spark-defaults.conf and sparv-env.sh that refer to IP or Master config, just do only this: On the master, in spark-env.sh, set the SPARK_MASTER_IP to the IP address, not hostname. Then use IP addresses in your call to Spark Context. See what happens. I know what you are seeing is two different bits of code working differently but I would bet it's an underlying Spark config issue. The important part is the master log which clearly identifies a network problem. As noted in my SO post, there's a bug out there that leads me to always use IP addresses but I am not sure how widely applicable that answer is :) If that doesn't work, please post what is the different between "WordCount MapReduce job" and "Spark Wordcount" -- that's not clear to me. Post your SparkConf and Spark Context calls. JimL I'm new to Spark. Before I describe the problem, I'd like to let you know the role of the machines that organize the cluster and the purpose of my work. By reading and follwing the instructions and tutorials, I successfully built up a cluster with 7 CentOS-6.5 machines. I installed Hadoop 2.7.1, Spark 1.5.1, Scala 2.10.4 and ZooKeeper 3.4.5 on them. The details are listed as below: As all the other guys in our group are in the habit of eclipse on Windows, I'm trying to work on this. I have successfully submitted the WordCount MapReduce job to YARN and it run smoothly through eclipse and Windows. But when I tried to run the Spark WordCount, it gives me the following error in the eclipse console: ... 15/12/23 11:15:33 ERROR ErrorMonitor: dropping message [class akka.actor.ActorSelectionMessage] for non-local recipient [Actor[akka.tcp://sparkMaster@10.20.17.70:7077/]] arriving at [akka.tcp://sparkMaster@10.20.17.70:7077] inbound addresses are [akka.tcp://sparkMaster@hadoop00:7077] akka.event.Logging$Error$NoCause$ 15/12/23 11:15:53 INFO Master: 10.20.6.23:56374 got disassociated, removing it. 15/12/23 11:15:53 INFO Master: 10.20.6.23:56374 got disassociated, removing it. 15/12/23 11:15:53 WARN ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkDriver@10.20.6.23:56374] has failed, address is now gated for [5000] ms. Reason: [Disassociated] ... object WordCount{ def main(args: Array[String]){ val conf = new SparkConf().setAppName("Scala WordCount").setMaster("spark://10.20.17.70:7077").setJars(List("C:\\Temp\\test.jar")); val sc = new SparkContext(conf); val textFile = sc.textFile("hdfs://10.20.17.70:9000/wc/indata/wht.txt"); textFile.flatMap(_.split(" ")).map((_, 1)).reduceByKey(_+_).collect().foreach(println); } } -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Problem-of-submitting-Spark-task-to-cluster-from-eclipse-IDE-on-Windows-tp25778p25825.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
