This issue is related to your cluster. Can you paste your spark-env.sh? Also you can try starting a spark-shell like $SPARK_HOME/bin/spark-shell --master=*spark://ip-172-31-24-183.ec2.internal:7077* and try the same code in it.
Is this the same URI spark://ip-172-31-24-183.ec2.internal:7077 shown in your webUI http://master-host:8080(top left)? Thanks Best Regards On Tue, Oct 7, 2014 at 8:39 PM, Jacob Chacko - Catalyst Consulting < [email protected]> wrote: > Hi All > > I have one master and one worker on AWS (amazon web service) and am > running spark map reduce code provided on the link > https://spark.apache.org/examples.html > We are using Spark version 1.0.2 > > Word Count > val file = spark.textFile("hdfs://...") > val counts = file.flatMap(line => line.split(" ")) > .map(word => (word, 1)) > .reduceByKey(_ + _) > counts.saveAsTextFile("hdfs://...") > > To get this working I have compiled a jar file. > spark-submit --master spark://ip-172-31-24-183.ec2.internal:7077 --class > Sample /home/ec2-user/scalatest/target/scala-2.11/test_big_2.11-0.1.jar > > > when I run the spark job it gives me error: Connection refused - > > > 1. There is not result shown on the master > 2. Error on worker is as below > > > 14/10/07 10:16:16 ERROR EndpointWriter: AssociationError > [akka.tcp://[email protected]:39142] -> > [akka.tcp://[email protected]:38752]: Error > [Association failed with > [akka.tcp://[email protected]:38752]] [ > akka.remote.EndpointAssociationException: Association failed with > [akka.tcp://[email protected]:38752] > Caused by: > akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: > Connection refused: ip-172-31-27-51.ec2.internal/172.31.27.51:38752 > ] > > Could some one help clarify if this is a > > - spark issue > - issue with the jar file > - or if slave cannot connect to master or itself or some port ? > > > Please help > > Thanks > Jacob > > > >
