Re: Shuffle service fails to register driver - Spark - Mesos

2016-06-22 Thread Feller, Eugen 
Make sure you are running the MesosShuffleService and not the standard shuffle 
service:

 *   org.apache.spark.deploy.mesos.MesosExternalShuffleService v.s. 
org.apache.spark.deploy.ExternalShuffleService
 *   start-mesos-shuffle-service.sh v.s. start-shuffle-service.sh

Thanks to Timothy Chen for the hint.

- Eugen Feller

Re: Shuffle service fails to register driver - Spark - Mesos

2016-04-15 Thread Jo Voordeckers 
Never mind, just figured out my problem,

I was running: *deploy.ExternalShuffleService* instead of
*deploy.mesos.MesosExternalShuffleService*

- Jo Voordeckers


On Fri, Apr 15, 2016 at 2:29 PM, Jo Voordeckers 
wrote:

> Forgot to mention we're running Spark (Streaming) 1.5.1
>
> - Jo Voordeckers
>
>
> On Fri, Apr 15, 2016 at 12:21 PM, Jo Voordeckers  > wrote:
>
>> Hi all,
>>
>> I've got mesos in coarse grained mode with dyn alloc, shuffle service
>> enabled and am running the shuffle service on every mesos slave.
>>
>> I'm assuming I misconfigured something on the scheduler service, any
>> ideas?
>>
>> On my driver is see a few of these, I guess it's one for every executor :
>>
>> 19:12:29 WARN [shuffle-client-1] MesosExternalShuffleClient - Unable to
>>> register app 78a944c9-3a89-4334-bca3-7108aadb1798- with external
>>> shuffle service. Please manually remove shuffle data after driver exit.
>>> Error: java.lang.RuntimeException: java.lang.UnsupportedOperationException:
>>> Unexpected message:
>>> org.apache.spark.network.shuffle.protocol.mesos.RegisterDriver@c399c59a
>>> at
>>> org.apache.spark.network.shuffle.ExternalShuffleBlockHandler.handleMessage(ExternalShuffleBlockHandler.java:92)
>>> at
>>> org.apache.spark.network.shuffle.ExternalShuffleBlockHandler.receive(ExternalShuffleBlockHandler.java:68)
>>> at
>>> org.apache.spark.network.server.TransportRequestHandler.processRpcRequest(TransportRequestHandler.java:114)
>>> at
>>> org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:87)
>>> at
>>> org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:101)
>>> at
>>> org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:51)
>>> at
>>> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
>>> at
>>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
>>> at
>>> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
>>>
>> [...]
>>
>> In the scheduler service I see logs like this on every box:
>>
>> log4j:WARN No appenders could be found for logger
>>> (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
>>> log4j:WARN Please initialize the log4j system properly.
>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>> for more info.
>>> Using Spark's repl log4j profile:
>>> org/apache/spark/log4j-defaults-repl.properties
>>> To adjust logging level use sc.setLogLevel("INFO")
>>> 16/04/14 19:12:29 ERROR TransportRequestHandler: Error while invoking
>>> RpcHandler#receive() on RPC id 7280403447531815366
>>> java.lang.UnsupportedOperationException: Unexpected message:
>>> org.apache.spark.network.shuffle.protocol.mesos.RegisterDriver@c399c59a
>>> at
>>> org.apache.spark.network.shuffle.ExternalShuffleBlockHandler.handleMessage(ExternalShuffleBlockHandler.java:92)
>>> at
>>> org.apache.spark.network.shuffle.ExternalShuffleBlockHandler.receive(ExternalShuffleBlockHandler.java:68)
>>> at
>>> org.apache.spark.network.server.TransportRequestHandler.processRpcRequest(TransportRequestHandler.java:114)
>>> at
>>> org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:87)
>>> at
>>> org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:101)
>>> at
>>> org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:51)
>>> at
>>> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
>>> at
>>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
>>>
>> [...]
>>
>> Thanks!
>>
>> - Jo Voordeckers
>>
>>
>

Re: Shuffle service fails to register driver - Spark - Mesos

2016-04-15 Thread Jo Voordeckers 
Forgot to mention we're running Spark (Streaming) 1.5.1

- Jo Voordeckers


On Fri, Apr 15, 2016 at 12:21 PM, Jo Voordeckers 
wrote:

> Hi all,
>
> I've got mesos in coarse grained mode with dyn alloc, shuffle service
> enabled and am running the shuffle service on every mesos slave.
>
> I'm assuming I misconfigured something on the scheduler service, any ideas?
>
> On my driver is see a few of these, I guess it's one for every executor :
>
> 19:12:29 WARN [shuffle-client-1] MesosExternalShuffleClient - Unable to
>> register app 78a944c9-3a89-4334-bca3-7108aadb1798- with external
>> shuffle service. Please manually remove shuffle data after driver exit.
>> Error: java.lang.RuntimeException: java.lang.UnsupportedOperationException:
>> Unexpected message:
>> org.apache.spark.network.shuffle.protocol.mesos.RegisterDriver@c399c59a
>> at
>> org.apache.spark.network.shuffle.ExternalShuffleBlockHandler.handleMessage(ExternalShuffleBlockHandler.java:92)
>> at
>> org.apache.spark.network.shuffle.ExternalShuffleBlockHandler.receive(ExternalShuffleBlockHandler.java:68)
>> at
>> org.apache.spark.network.server.TransportRequestHandler.processRpcRequest(TransportRequestHandler.java:114)
>> at
>> org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:87)
>> at
>> org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:101)
>> at
>> org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:51)
>> at
>> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
>> at
>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
>> at
>> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
>>
> [...]
>
> In the scheduler service I see logs like this on every box:
>
> log4j:WARN No appenders could be found for logger
>> (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
>> log4j:WARN Please initialize the log4j system properly.
>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
>> more info.
>> Using Spark's repl log4j profile:
>> org/apache/spark/log4j-defaults-repl.properties
>> To adjust logging level use sc.setLogLevel("INFO")
>> 16/04/14 19:12:29 ERROR TransportRequestHandler: Error while invoking
>> RpcHandler#receive() on RPC id 7280403447531815366
>> java.lang.UnsupportedOperationException: Unexpected message:
>> org.apache.spark.network.shuffle.protocol.mesos.RegisterDriver@c399c59a
>> at
>> org.apache.spark.network.shuffle.ExternalShuffleBlockHandler.handleMessage(ExternalShuffleBlockHandler.java:92)
>> at
>> org.apache.spark.network.shuffle.ExternalShuffleBlockHandler.receive(ExternalShuffleBlockHandler.java:68)
>> at
>> org.apache.spark.network.server.TransportRequestHandler.processRpcRequest(TransportRequestHandler.java:114)
>> at
>> org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:87)
>> at
>> org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:101)
>> at
>> org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:51)
>> at
>> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
>> at
>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
>>
> [...]
>
> Thanks!
>
> - Jo Voordeckers
>
>

Shuffle service fails to register driver - Spark - Mesos

2016-04-15 Thread Jo Voordeckers 
Hi all,

I've got mesos in coarse grained mode with dyn alloc, shuffle service
enabled and am running the shuffle service on every mesos slave.

I'm assuming I misconfigured something on the scheduler service, any ideas?

On my driver is see a few of these, I guess it's one for every executor :

19:12:29 WARN [shuffle-client-1] MesosExternalShuffleClient - Unable to
> register app 78a944c9-3a89-4334-bca3-7108aadb1798- with external
> shuffle service. Please manually remove shuffle data after driver exit.
> Error: java.lang.RuntimeException: java.lang.UnsupportedOperationException:
> Unexpected message:
> org.apache.spark.network.shuffle.protocol.mesos.RegisterDriver@c399c59a
> at
> org.apache.spark.network.shuffle.ExternalShuffleBlockHandler.handleMessage(ExternalShuffleBlockHandler.java:92)
> at
> org.apache.spark.network.shuffle.ExternalShuffleBlockHandler.receive(ExternalShuffleBlockHandler.java:68)
> at
> org.apache.spark.network.server.TransportRequestHandler.processRpcRequest(TransportRequestHandler.java:114)
> at
> org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:87)
> at
> org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:101)
> at
> org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:51)
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
>
[...]

In the scheduler service I see logs like this on every box:

log4j:WARN No appenders could be found for logger
> (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
> more info.
> Using Spark's repl log4j profile:
> org/apache/spark/log4j-defaults-repl.properties
> To adjust logging level use sc.setLogLevel("INFO")
> 16/04/14 19:12:29 ERROR TransportRequestHandler: Error while invoking
> RpcHandler#receive() on RPC id 7280403447531815366
> java.lang.UnsupportedOperationException: Unexpected message:
> org.apache.spark.network.shuffle.protocol.mesos.RegisterDriver@c399c59a
> at
> org.apache.spark.network.shuffle.ExternalShuffleBlockHandler.handleMessage(ExternalShuffleBlockHandler.java:92)
> at
> org.apache.spark.network.shuffle.ExternalShuffleBlockHandler.receive(ExternalShuffleBlockHandler.java:68)
> at
> org.apache.spark.network.server.TransportRequestHandler.processRpcRequest(TransportRequestHandler.java:114)
> at
> org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:87)
> at
> org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:101)
> at
> org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:51)
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
>
[...]

Thanks!

- Jo Voordeckers