Re: keep or remove sc.stop() coz of RpcEnv already stopped error

2017-03-13 Thread Alex 
Hi ,
I am using spark-1.6 how to ignore this warning because of this Illegal
state exception my production jobs which are scheduld are showing completed
abnormally... I cant even handle exception as after sc.stop if i try to
execute any code again this exception will come from catch block.. so i
removed sc.stop completely and let it exit on its own is it okay?

On Mon, Mar 13, 2017 at 7:01 PM, Yong Zhang  wrote:

> What version of Spark you are using?
>
>
> Based on Spark-12967, it is fixed on Spark 2.0 and later. If you are using
> Spark 1.x, you can ignore this Warning. It shouldn't affect any functions.
>
>
> Yong
>
> --
> *From:* nancy henry 
> *Sent:* Monday, March 13, 2017 7:08 AM
> *To:* user @spark
> *Subject:* keep or remove sc.stop() coz of RpcEnv already stopped error
>
>
>  Hi Team,
>
>
> getting this error if we put sc.stop() in application..
>
>
>
> can we remove it from application but i read if yu dont explicitly stop
> using sc.stop the yarn application will not get registered in history
> service.. SO what to do ?
>
>
>  WARN Dispatcher: Message RemoteProcessDisconnected dropped.
>
> java.lang.IllegalStateException: RpcEnv already stopped.
>
> at org.apache.spark.rpc.netty.Dispatcher.postMessage(
> Dispatcher.scala:159)
>
> at org.apache.spark.rpc.netty.Dispatcher.postToAll(
> Dispatcher.scala:109)
>
> at org.apache.spark.rpc.netty.NettyRpcHandler.
> connectionTerminated(NettyRpcEnv.scala:630)
>
> at org.apache.spark.network.server.TransportRequestHandler.
> channelUnregistered(TransportRequestHandler.java:94)
>
>
>
>
>

Re: keep or remove sc.stop() coz of RpcEnv already stopped error

2017-03-13 Thread Yong Zhang 
What version of Spark you are using?


Based on Spark-12967, it is fixed on Spark 2.0 and later. If you are using 
Spark 1.x, you can ignore this Warning. It shouldn't affect any functions.


Yong


From: nancy henry 
Sent: Monday, March 13, 2017 7:08 AM
To: user @spark
Subject: keep or remove sc.stop() coz of RpcEnv already stopped error

 Hi Team,

getting this error if we put sc.stop() in application..

can we remove it from application but i read if yu dont explicitly stop using 
sc.stop the yarn application will not get registered in history service.. SO 
what to do ?

 WARN Dispatcher: Message RemoteProcessDisconnected dropped.
java.lang.IllegalStateException: RpcEnv already stopped.
at 
org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:159)
at org.apache.spark.rpc.netty.Dispatcher.postToAll(Dispatcher.scala:109)
at 
org.apache.spark.rpc.netty.NettyRpcHandler.connectionTerminated(NettyRpcEnv.scala:630)
at 
org.apache.spark.network.server.TransportRequestHandler.channelUnregistered(TransportRequestHandler.java:94)

keep or remove sc.stop() coz of RpcEnv already stopped error

2017-03-13 Thread nancy henry 
 Hi Team,


getting this error if we put sc.stop() in application..



can we remove it from application but i read if yu dont explicitly stop
using sc.stop the yarn application will not get registered in history
service.. SO what to do ?


 WARN Dispatcher: Message RemoteProcessDisconnected dropped.

java.lang.IllegalStateException: RpcEnv already stopped.

at
org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:159)

at
org.apache.spark.rpc.netty.Dispatcher.postToAll(Dispatcher.scala:109)

at
org.apache.spark.rpc.netty.NettyRpcHandler.connectionTerminated(NettyRpcEnv.scala:630)

at
org.apache.spark.network.server.TransportRequestHandler.channelUnregistered(TransportRequestHandler.java:94)