Hi Andrew You are correct I am using cluster mode.
Many thanks Andy From: Andrew Or <and...@databricks.com> Date: Thursday, December 10, 2015 at 6:31 PM To: Andrew Davidson <a...@santacruzintegration.com> Cc: Jakob Odersky <joder...@gmail.com>, "user @spark" <user@spark.apache.org> Subject: Re: Warning: Master endpoint spark://ip:7077 was not a REST server. Falling back to legacy submission gateway instead. > Hi Andy, > > You must be running in cluster mode. The Spark Master accepts client mode > submissions on port 7077 and cluster mode submissions on port 6066. This is > because standalone cluster mode uses a REST API to submit applications by > default. If you submit to port 6066 instead the warning should go away. > > -Andrew > > > 2015-12-10 18:13 GMT-08:00 Andy Davidson <a...@santacruzintegration.com>: >> Hi Jakob >> >> The cluster was set up using the spark-1.5.1-bin-hadoop2.6/ec2/spark-ec2 >> script >> >> Given my limited knowledge I think this looks okay? >> >> Thanks >> >> Andy >> >> $ sudo netstat -peant | grep 7077 >> >> tcp 0 0 ::ffff:172-31-30-51:7077 :::* >> LISTEN 0 3116414 27355/java >> >> tcp 0 0 ::ffff:172-31-30-51:7077 ::ffff:172-31-30-51:57311 >> ESTABLISHED 0 3115919 27355/java >> >> tcp 0 0 ::ffff:172-31-30-51:7077 ::ffff:172-31-30-51:42333 >> ESTABLISHED 0 3736664 27355/java >> >> tcp 0 0 ::ffff:172-31-30-51:7077 ::ffff:172-31-30-51:49796 >> ESTABLISHED 0 3115925 27355/java >> >> tcp 0 0 ::ffff:172-31-30-51:7077 ::ffff:172-31-30-51:42290 >> ESTABLISHED 0 3115923 27355/java >> >> >> >> $ ps -aux | grep 27355 >> >> Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.8/FAQ >> >> ec2-user 23867 0.0 0.0 110404 872 pts/0 S+ 02:06 0:00 grep 27355 >> >> root 27355 0.5 6.7 3679096 515836 ? Sl Nov26 107:04 >> /usr/java/latest/bin/java -cp >> /root/spark/sbin/../conf/:/root/spark/lib/spark-assembly-1.5.1-hadoop1.2.1.ja >> r:/root/spark/lib/datanucleus-api-jdo-3.2.6.jar:/root/spark/lib/datanucleus-r >> dbms-3.2.9.jar:/root/spark/lib/datanucleus-core-3.2.10.jar:/root/ephemeral-hd >> fs/conf/ -Xms1g -Xmx1g org.apache.spark.deploy.master.Master --ip >> ec2-54-215-217-122.us-west-1.compute.amazonaws.com >> <http://ec2-54-215-217-122.us-west-1.compute.amazonaws.com> --port 7077 >> --webui-port 8080 >> >> >> From: Jakob Odersky <joder...@gmail.com> >> Date: Thursday, December 10, 2015 at 5:55 PM >> To: Andrew Davidson <a...@santacruzintegration.com> >> Cc: "user @spark" <user@spark.apache.org> >> Subject: Re: Warning: Master endpoint spark://ip:7077 was not a REST server. >> Falling back to legacy submission gateway instead. >> >>> Is there any other process using port 7077? >>> >>> On 10 December 2015 at 08:52, Andy Davidson <a...@santacruzintegration.com> >>> wrote: >>>> Hi >>>> >>>> I am using spark-1.5.1-bin-hadoop2.6. Any idea why I get this warning. My >>>> job seems to run with out any problem. >>>> >>>> Kind regards >>>> >>>> Andy >>>> >>>> + /root/spark/bin/spark-submit --class com.pws.spark.streaming.IngestDriver >>>> --master spark://ec2-54-205-209-122.us-west-1.compute.amazonaws.com:7077 >>>> <http://ec2-54-205-209-122.us-west-1.compute.amazonaws.com:7077> >>>> --total-executor-cores 2 --deploy-mode cluster >>>> hdfs:///home/ec2-user/build/ingest-all.jar --clusterMode --dirPath week_3 >>>> >>>> Running Spark using the REST application submission protocol. >>>> >>>> 15/12/10 16:46:33 WARN RestSubmissionClient: Unable to connect to server >>>> spark://ec2-54-205-209-122.us-west-1.compute.amazonaws.com:7077 >>>> <http://ec2-54-205-209-122.us-west-1.compute.amazonaws.com:7077> . >>>> >>>> Warning: Master endpoint >>>> ec2-54-205-209-122.us-west-1.compute.amazonaws.com:7077 >>>> <http://ec2-54-205-209-122.us-west-1.compute.amazonaws.com:7077> was not a >>>> REST server. Falling back to legacy submission gateway instead. >>> >
