Hi all, I've got mesos in coarse grained mode with dyn alloc, shuffle service enabled and am running the shuffle service on every mesos slave.
I'm assuming I misconfigured something on the scheduler service, any ideas? On my driver is see a few of these, I guess it's one for every executor : 19:12:29 WARN [shuffle-client-1] MesosExternalShuffleClient - Unable to > register app 78a944c9-3a89-4334-bca3-7108aadb1798-0000 with external > shuffle service. Please manually remove shuffle data after driver exit. > Error: java.lang.RuntimeException: java.lang.UnsupportedOperationException: > Unexpected message: > org.apache.spark.network.shuffle.protocol.mesos.RegisterDriver@c399c59a > at > org.apache.spark.network.shuffle.ExternalShuffleBlockHandler.handleMessage(ExternalShuffleBlockHandler.java:92) > at > org.apache.spark.network.shuffle.ExternalShuffleBlockHandler.receive(ExternalShuffleBlockHandler.java:68) > at > org.apache.spark.network.server.TransportRequestHandler.processRpcRequest(TransportRequestHandler.java:114) > at > org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:87) > at > org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:101) > at > org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:51) > at > io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294) > [...] In the scheduler service I see logs like this on every box: log4j:WARN No appenders could be found for logger > (org.apache.hadoop.metrics2.lib.MutableMetricsFactory). > log4j:WARN Please initialize the log4j system properly. > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for > more info. > Using Spark's repl log4j profile: > org/apache/spark/log4j-defaults-repl.properties > To adjust logging level use sc.setLogLevel("INFO") > 16/04/14 19:12:29 ERROR TransportRequestHandler: Error while invoking > RpcHandler#receive() on RPC id 7280403447531815366 > java.lang.UnsupportedOperationException: Unexpected message: > org.apache.spark.network.shuffle.protocol.mesos.RegisterDriver@c399c59a > at > org.apache.spark.network.shuffle.ExternalShuffleBlockHandler.handleMessage(ExternalShuffleBlockHandler.java:92) > at > org.apache.spark.network.shuffle.ExternalShuffleBlockHandler.receive(ExternalShuffleBlockHandler.java:68) > at > org.apache.spark.network.server.TransportRequestHandler.processRpcRequest(TransportRequestHandler.java:114) > at > org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:87) > at > org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:101) > at > org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:51) > at > io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308) > [...] Thanks! - Jo Voordeckers
