cluster mode uses port 6066 Re: Warning: Master endpoint spark://ip:7077 was not a REST server. Falling back to legacy submission gateway instead.

2015-12-11 Thread Andy Davidson 
Hi Andrew

You are correct I am using cluster mode.

Many thanks

Andy

From:  Andrew Or <and...@databricks.com>
Date:  Thursday, December 10, 2015 at 6:31 PM
To:  Andrew Davidson <a...@santacruzintegration.com>
Cc:  Jakob Odersky <joder...@gmail.com>, "user @spark"
<user@spark.apache.org>
Subject:  Re: Warning: Master endpoint spark://ip:7077 was not a REST
server. Falling back to legacy submission gateway instead.

> Hi Andy,
> 
> You must be running in cluster mode. The Spark Master accepts client mode
> submissions on port 7077 and cluster mode submissions on port 6066. This is
> because standalone cluster mode uses a REST API to submit applications by
> default. If you submit to port 6066 instead the warning should go away.
> 
> -Andrew
> 
> 
> 2015-12-10 18:13 GMT-08:00 Andy Davidson <a...@santacruzintegration.com>:
>> Hi Jakob
>> 
>> The cluster was set up using the spark-1.5.1-bin-hadoop2.6/ec2/spark-ec2
>> script
>> 
>> Given my limited knowledge I think this looks okay?
>> 
>> Thanks
>> 
>> Andy
>> 
>> $ sudo netstat -peant | grep 7077
>> 
>> tcp0  0 :::172-31-30-51:7077:::*
>> LISTEN  0  311641427355/java
>> 
>> tcp0  0 :::172-31-30-51:7077:::172-31-30-51:57311
>> ESTABLISHED 0  311591927355/java
>> 
>> tcp0  0 :::172-31-30-51:7077:::172-31-30-51:42333
>> ESTABLISHED 0  373666427355/java
>> 
>> tcp0  0 :::172-31-30-51:7077:::172-31-30-51:49796
>> ESTABLISHED 0  311592527355/java
>> 
>> tcp0  0 :::172-31-30-51:7077:::172-31-30-51:42290
>> ESTABLISHED 0  311592327355/java
>> 
>> 
>> 
>> $ ps -aux | grep 27355
>> 
>> Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.8/FAQ
>> 
>> ec2-user 23867  0.0  0.0 110404   872 pts/0S+   02:06   0:00 grep 27355
>> 
>> root 27355  0.5  6.7 3679096 515836 ?  Sl   Nov26 107:04
>> /usr/java/latest/bin/java -cp
>> /root/spark/sbin/../conf/:/root/spark/lib/spark-assembly-1.5.1-hadoop1.2.1.ja
>> r:/root/spark/lib/datanucleus-api-jdo-3.2.6.jar:/root/spark/lib/datanucleus-r
>> dbms-3.2.9.jar:/root/spark/lib/datanucleus-core-3.2.10.jar:/root/ephemeral-hd
>> fs/conf/ -Xms1g -Xmx1g org.apache.spark.deploy.master.Master --ip
>> ec2-54-215-217-122.us-west-1.compute.amazonaws.com
>> <http://ec2-54-215-217-122.us-west-1.compute.amazonaws.com>  --port 7077
>> --webui-port 8080
>> 
>> 
>> From:  Jakob Odersky <joder...@gmail.com>
>> Date:  Thursday, December 10, 2015 at 5:55 PM
>> To:  Andrew Davidson <a...@santacruzintegration.com>
>> Cc:  "user @spark" <user@spark.apache.org>
>> Subject:  Re: Warning: Master endpoint spark://ip:7077 was not a REST server.
>> Falling back to legacy submission gateway instead.
>> 
>>> Is there any other process using port 7077?
>>> 
>>> On 10 December 2015 at 08:52, Andy Davidson <a...@santacruzintegration.com>
>>> wrote:
>>>> Hi
>>>> 
>>>> I am using spark-1.5.1-bin-hadoop2.6. Any idea why I get this warning. My
>>>> job seems to run with out any problem.
>>>> 
>>>> Kind regards 
>>>> 
>>>> Andy
>>>> 
>>>> + /root/spark/bin/spark-submit --class com.pws.spark.streaming.IngestDriver
>>>> --master spark://ec2-54-205-209-122.us-west-1.compute.amazonaws.com:7077
>>>> <http://ec2-54-205-209-122.us-west-1.compute.amazonaws.com:7077>
>>>> --total-executor-cores 2 --deploy-mode cluster
>>>> hdfs:///home/ec2-user/build/ingest-all.jar --clusterMode --dirPath week_3
>>>> 
>>>> Running Spark using the REST application submission protocol.
>>>> 
>>>> 15/12/10 16:46:33 WARN RestSubmissionClient: Unable to connect to server
>>>> spark://ec2-54-205-209-122.us-west-1.compute.amazonaws.com:7077
>>>> <http://ec2-54-205-209-122.us-west-1.compute.amazonaws.com:7077> .
>>>> 
>>>> Warning: Master endpoint
>>>> ec2-54-205-209-122.us-west-1.compute.amazonaws.com:7077
>>>> <http://ec2-54-205-209-122.us-west-1.compute.amazonaws.com:7077>  was not a
>>>> REST server. Falling back to legacy submission gateway instead.
>>> 
>

Re: Warning: Master endpoint spark://ip:7077 was not a REST server. Falling back to legacy submission gateway instead.

2015-12-10 Thread Jakob Odersky 
Is there any other process using port 7077?

On 10 December 2015 at 08:52, Andy Davidson 
wrote:

> Hi
>
> I am using spark-1.5.1-bin-hadoop2.6. Any idea why I get this warning. My
> job seems to run with out any problem.
>
> Kind regards
>
> Andy
>
> + /root/spark/bin/spark-submit --class
> com.pws.spark.streaming.IngestDriver --master spark://
> ec2-54-205-209-122.us-west-1.compute.amazonaws.com:7077
> --total-executor-cores 2 --deploy-mode cluster
> hdfs:///home/ec2-user/build/ingest-all.jar --clusterMode --dirPath week_3
>
> Running Spark using the REST application submission protocol.
>
> 15/12/10 16:46:33 WARN RestSubmissionClient: Unable to connect to server
> spark://ec2-54-205-209-122.us-west-1.compute.amazonaws.com:7077.
>
> Warning: Master endpoint
> ec2-54-205-209-122.us-west-1.compute.amazonaws.com:7077 was not a REST
> server. Falling back to legacy submission gateway instead.
>

Re: Warning: Master endpoint spark://ip:7077 was not a REST server. Falling back to legacy submission gateway instead.

2015-12-10 Thread Andy Davidson 
Hi Jakob

The cluster was set up using the spark-1.5.1-bin-hadoop2.6/ec2/spark-ec2
script

Given my limited knowledge I think this looks okay?

Thanks

Andy

$ sudo netstat -peant | grep 7077

tcp0  0 :::172-31-30-51:7077:::*
LISTEN  0  311641427355/java

tcp0  0 :::172-31-30-51:7077:::172-31-30-51:57311
ESTABLISHED 0  311591927355/java

tcp0  0 :::172-31-30-51:7077:::172-31-30-51:42333
ESTABLISHED 0  373666427355/java

tcp0  0 :::172-31-30-51:7077:::172-31-30-51:49796
ESTABLISHED 0  311592527355/java

tcp0  0 :::172-31-30-51:7077:::172-31-30-51:42290
ESTABLISHED 0  311592327355/java



$ ps -aux | grep 27355

Warning: bad syntax, perhaps a bogus '-'? See
/usr/share/doc/procps-3.2.8/FAQ

ec2-user 23867  0.0  0.0 110404   872 pts/0S+   02:06   0:00 grep 27355

root 27355  0.5  6.7 3679096 515836 ?  Sl   Nov26 107:04
/usr/java/latest/bin/java -cp
/root/spark/sbin/../conf/:/root/spark/lib/spark-assembly-1.5.1-hadoop1.2.1.j
ar:/root/spark/lib/datanucleus-api-jdo-3.2.6.jar:/root/spark/lib/datanucleus
-rdbms-3.2.9.jar:/root/spark/lib/datanucleus-core-3.2.10.jar:/root/ephemeral
-hdfs/conf/ -Xms1g -Xmx1g org.apache.spark.deploy.master.Master --ip
ec2-54-215-217-122.us-west-1.compute.amazonaws.com --port 7077 --webui-port
8080


From:  Jakob Odersky <joder...@gmail.com>
Date:  Thursday, December 10, 2015 at 5:55 PM
To:  Andrew Davidson <a...@santacruzintegration.com>
Cc:  "user @spark" <user@spark.apache.org>
Subject:  Re: Warning: Master endpoint spark://ip:7077 was not a REST
server. Falling back to legacy submission gateway instead.

> Is there any other process using port 7077?
> 
> On 10 December 2015 at 08:52, Andy Davidson <a...@santacruzintegration.com>
> wrote:
>> Hi
>> 
>> I am using spark-1.5.1-bin-hadoop2.6. Any idea why I get this warning. My job
>> seems to run with out any problem.
>> 
>> Kind regards 
>> 
>> Andy
>> 
>> + /root/spark/bin/spark-submit --class com.pws.spark.streaming.IngestDriver
>> --master spark://ec2-54-205-209-122.us-west-1.compute.amazonaws.com:7077
>> <http://ec2-54-205-209-122.us-west-1.compute.amazonaws.com:7077>
>> --total-executor-cores 2 --deploy-mode cluster
>> hdfs:///home/ec2-user/build/ingest-all.jar --clusterMode --dirPath week_3
>> 
>> Running Spark using the REST application submission protocol.
>> 
>> 15/12/10 16:46:33 WARN RestSubmissionClient: Unable to connect to server
>> spark://ec2-54-205-209-122.us-west-1.compute.amazonaws.com:7077
>> <http://ec2-54-205-209-122.us-west-1.compute.amazonaws.com:7077> .
>> 
>> Warning: Master endpoint
>> ec2-54-205-209-122.us-west-1.compute.amazonaws.com:7077
>> <http://ec2-54-205-209-122.us-west-1.compute.amazonaws.com:7077>  was not a
>> REST server. Falling back to legacy submission gateway instead.
>

Re: Warning: Master endpoint spark://ip:7077 was not a REST server. Falling back to legacy submission gateway instead.

2015-12-10 Thread Andrew Or 
Hi Andy,

You must be running in cluster mode. The Spark Master accepts client mode
submissions on port 7077 and cluster mode submissions on port 6066. This is
because standalone cluster mode uses a REST API to submit applications by
default. If you submit to port 6066 instead the warning should go away.

-Andrew


2015-12-10 18:13 GMT-08:00 Andy Davidson <a...@santacruzintegration.com>:

> Hi Jakob
>
> The cluster was set up using the spark-1.5.1-bin-hadoop2.6/ec2/spark-ec2
> script
>
> Given my limited knowledge I think this looks okay?
>
> Thanks
>
> Andy
>
> $ sudo netstat -peant | grep 7077
>
> tcp0  0 :::172-31-30-51:7077:::*
>  LISTEN  0  311641427355/java
>
> tcp0  0 :::172-31-30-51:7077:::172-31-30-51:57311
> ESTABLISHED 0  311591927355/java
>
> tcp0  0 :::172-31-30-51:7077:::172-31-30-51:42333
> ESTABLISHED 0  373666427355/java
>
> tcp0  0 :::172-31-30-51:7077:::172-31-30-51:49796
> ESTABLISHED 0  311592527355/java
>
> tcp0  0 :::172-31-30-51:7077:::172-31-30-51:42290
> ESTABLISHED 0  311592327355/java
>
>
> $ ps -aux | grep 27355
>
> Warning: bad syntax, perhaps a bogus '-'? See
> /usr/share/doc/procps-3.2.8/FAQ
>
> ec2-user 23867  0.0  0.0 110404   872 pts/0S+   02:06   0:00 grep 27355
>
> root 27355  0.5  6.7 3679096 515836 ?  Sl   Nov26 107:04
> /usr/java/latest/bin/java -cp
> /root/spark/sbin/../conf/:/root/spark/lib/spark-assembly-1.5.1-hadoop1.2.1.jar:/root/spark/lib/datanucleus-api-jdo-3.2.6.jar:/root/spark/lib/datanucleus-rdbms-3.2.9.jar:/root/spark/lib/datanucleus-core-3.2.10.jar:/root/ephemeral-hdfs/conf/
> -Xms1g -Xmx1g org.apache.spark.deploy.master.Master --ip
> ec2-54-215-217-122.us-west-1.compute.amazonaws.com --port 7077
> --webui-port 8080
>
> From: Jakob Odersky <joder...@gmail.com>
> Date: Thursday, December 10, 2015 at 5:55 PM
> To: Andrew Davidson <a...@santacruzintegration.com>
> Cc: "user @spark" <user@spark.apache.org>
> Subject: Re: Warning: Master endpoint spark://ip:7077 was not a REST
> server. Falling back to legacy submission gateway instead.
>
> Is there any other process using port 7077?
>
> On 10 December 2015 at 08:52, Andy Davidson <a...@santacruzintegration.com
> > wrote:
>
>> Hi
>>
>> I am using spark-1.5.1-bin-hadoop2.6. Any idea why I get this warning.
>> My job seems to run with out any problem.
>>
>> Kind regards
>>
>> Andy
>>
>> + /root/spark/bin/spark-submit --class
>> com.pws.spark.streaming.IngestDriver --master spark://
>> ec2-54-205-209-122.us-west-1.compute.amazonaws.com:7077
>> --total-executor-cores 2 --deploy-mode cluster
>> hdfs:///home/ec2-user/build/ingest-all.jar --clusterMode --dirPath week_3
>>
>> Running Spark using the REST application submission protocol.
>>
>> 15/12/10 16:46:33 WARN RestSubmissionClient: Unable to connect to server
>> spark://ec2-54-205-209-122.us-west-1.compute.amazonaws.com:7077.
>>
>> Warning: Master endpoint
>> ec2-54-205-209-122.us-west-1.compute.amazonaws.com:7077 was not a REST
>> server. Falling back to legacy submission gateway instead.
>>
>
>

Warning: Master endpoint spark://ip:7077 was not a REST server. Falling back to legacy submission gateway instead.

2015-12-10 Thread Andy Davidson 
Hi

I am using spark-1.5.1-bin-hadoop2.6. Any idea why I get this warning. My
job seems to run with out any problem.

Kind regards 

Andy

+ /root/spark/bin/spark-submit --class com.pws.spark.streaming.IngestDriver
--master spark://ec2-54-205-209-122.us-west-1.compute.amazonaws.com:7077
--total-executor-cores 2 --deploy-mode cluster
hdfs:///home/ec2-user/build/ingest-all.jar --clusterMode --dirPath week_3

Running Spark using the REST application submission protocol.

15/12/10 16:46:33 WARN RestSubmissionClient: Unable to connect to server
spark://ec2-54-205-209-122.us-west-1.compute.amazonaws.com:7077.

Warning: Master endpoint
ec2-54-205-209-122.us-west-1.compute.amazonaws.com:7077 was not a REST
server. Falling back to legacy submission gateway instead.