Re: spark worker continuously trying to connect to master and failed in standalone mode

2016-07-22 Thread Neil Chang 
Thank you guys, it is the port issue.

On Wed, Jul 20, 2016 at 11:03 AM, Igor Berman  wrote:

> in addition check what ip the master is binding to(with nestat)
>
> On 20 July 2016 at 06:12, Andrew Ehrlich  wrote:
>
>> Troubleshooting steps:
>>
>> $ telnet localhost 7077 (on master, to confirm port is open)
>> $ telnet  7077 (on slave, to confirm port is blocked)
>>
>> If the port is available on the master from the master, but not on the
>> master from the slave, check firewall settings on the master:
>> https://help.ubuntu.com/lts/serverguide/firewall.html
>>
>> On Jul 19, 2016, at 6:25 PM, Neil Chang  wrote:
>>
>> Hi,
>>   I have two virtual pcs on private cloud (ubuntu 14). I installed spark
>> 2.0 preview on both machines. I then tried to test it with standalone mode.
>> I have no problem start the master. However, when I start the worker
>> (slave) on another machine, it makes many attempts to connect to master and
>> failed at the end.
>>   I can ssh from each machine to another without any problem. I can also
>> run a master and worker at the same machine without any problem.
>>
>> What did I miss? Any clue?
>>
>> here are the messages:
>>
>> WARN NativeCodeLoader: Unable to load native-hadoop library for your
>> platform ... using builtin-java classes where applicable
>> ..
>> INFO Worker: Connecting to master ip:7077 ...
>> INFO Worker: Retrying connection to master (attempt #1)
>> ..
>> INFO Worker: Retrying connection to master (attempt #7)
>> java.lang.IllegalArgumentException: requirement failed: TransportClient
>> has not yet been set.
>>at scala.Predef$.require(Predef.scala:224)
>> ...
>> WARN NettyRocEnv: Ignored failure: java.io.IOException: Connecting to
>> ip:7077 timed out
>> WARN Worker: Failed to connect to master ip.7077
>>
>>
>>
>> Thanks,
>> Neil
>>
>>
>>
>

Re: spark worker continuously trying to connect to master and failed in standalone mode

2016-07-20 Thread Igor Berman 
in addition check what ip the master is binding to(with nestat)

On 20 July 2016 at 06:12, Andrew Ehrlich  wrote:

> Troubleshooting steps:
>
> $ telnet localhost 7077 (on master, to confirm port is open)
> $ telnet  7077 (on slave, to confirm port is blocked)
>
> If the port is available on the master from the master, but not on the
> master from the slave, check firewall settings on the master:
> https://help.ubuntu.com/lts/serverguide/firewall.html
>
> On Jul 19, 2016, at 6:25 PM, Neil Chang  wrote:
>
> Hi,
>   I have two virtual pcs on private cloud (ubuntu 14). I installed spark
> 2.0 preview on both machines. I then tried to test it with standalone mode.
> I have no problem start the master. However, when I start the worker
> (slave) on another machine, it makes many attempts to connect to master and
> failed at the end.
>   I can ssh from each machine to another without any problem. I can also
> run a master and worker at the same machine without any problem.
>
> What did I miss? Any clue?
>
> here are the messages:
>
> WARN NativeCodeLoader: Unable to load native-hadoop library for your
> platform ... using builtin-java classes where applicable
> ..
> INFO Worker: Connecting to master ip:7077 ...
> INFO Worker: Retrying connection to master (attempt #1)
> ..
> INFO Worker: Retrying connection to master (attempt #7)
> java.lang.IllegalArgumentException: requirement failed: TransportClient
> has not yet been set.
>at scala.Predef$.require(Predef.scala:224)
> ...
> WARN NettyRocEnv: Ignored failure: java.io.IOException: Connecting to
> ip:7077 timed out
> WARN Worker: Failed to connect to master ip.7077
>
>
>
> Thanks,
> Neil
>
>
>

Re: spark worker continuously trying to connect to master and failed in standalone mode

2016-07-19 Thread Andrew Ehrlich 
Troubleshooting steps:

$ telnet localhost 7077 (on master, to confirm port is open)
$ telnet  7077 (on slave, to confirm port is blocked)

If the port is available on the master from the master, but not on the master 
from the slave, check firewall settings on the master: 
https://help.ubuntu.com/lts/serverguide/firewall.html 

> On Jul 19, 2016, at 6:25 PM, Neil Chang  wrote:
> 
> Hi,
>   I have two virtual pcs on private cloud (ubuntu 14). I installed spark 2.0 
> preview on both machines. I then tried to test it with standalone mode.
> I have no problem start the master. However, when I start the worker (slave) 
> on another machine, it makes many attempts to connect to master and failed at 
> the end. 
>   I can ssh from each machine to another without any problem. I can also run 
> a master and worker at the same machine without any problem.
> 
> What did I miss? Any clue?
> 
> here are the messages:
> 
> WARN NativeCodeLoader: Unable to load native-hadoop library for your platform 
> ... using builtin-java classes where applicable
> ..
> INFO Worker: Connecting to master ip:7077 ... 
> INFO Worker: Retrying connection to master (attempt #1)
> ..
> INFO Worker: Retrying connection to master (attempt #7)
> java.lang.IllegalArgumentException: requirement failed: TransportClient has 
> not yet been set.
>at scala.Predef$.require(Predef.scala:224)
> ...
> WARN NettyRocEnv: Ignored failure: java.io.IOException: Connecting to ip:7077 
> timed out
> WARN Worker: Failed to connect to master ip.7077
> 
> 
> 
> Thanks,
> Neil

spark worker continuously trying to connect to master and failed in standalone mode

2016-07-19 Thread Neil Chang 
Hi,
  I have two virtual pcs on private cloud (ubuntu 14). I installed spark
2.0 preview on both machines. I then tried to test it with standalone mode.
I have no problem start the master. However, when I start the worker
(slave) on another machine, it makes many attempts to connect to master and
failed at the end.
  I can ssh from each machine to another without any problem. I can also
run a master and worker at the same machine without any problem.

What did I miss? Any clue?

here are the messages:

WARN NativeCodeLoader: Unable to load native-hadoop library for your
platform ... using builtin-java classes where applicable
..
INFO Worker: Connecting to master ip:7077 ...
INFO Worker: Retrying connection to master (attempt #1)
..
INFO Worker: Retrying connection to master (attempt #7)
java.lang.IllegalArgumentException: requirement failed: TransportClient has
not yet been set.
   at scala.Predef$.require(Predef.scala:224)
...
WARN NettyRocEnv: Ignored failure: java.io.IOException: Connecting to
ip:7077 timed out
WARN Worker: Failed to connect to master ip.7077



Thanks,
Neil