Forgot to mention we're running Spark (Streaming) 1.5.1 - Jo Voordeckers
On Fri, Apr 15, 2016 at 12:21 PM, Jo Voordeckers <jo.voordeck...@gmail.com> wrote: > Hi all, > > I've got mesos in coarse grained mode with dyn alloc, shuffle service > enabled and am running the shuffle service on every mesos slave. > > I'm assuming I misconfigured something on the scheduler service, any ideas? > > On my driver is see a few of these, I guess it's one for every executor : > > 19:12:29 WARN [shuffle-client-1] MesosExternalShuffleClient - Unable to >> register app 78a944c9-3a89-4334-bca3-7108aadb1798-0000 with external >> shuffle service. Please manually remove shuffle data after driver exit. >> Error: java.lang.RuntimeException: java.lang.UnsupportedOperationException: >> Unexpected message: >> org.apache.spark.network.shuffle.protocol.mesos.RegisterDriver@c399c59a >> at >> org.apache.spark.network.shuffle.ExternalShuffleBlockHandler.handleMessage(ExternalShuffleBlockHandler.java:92) >> at >> org.apache.spark.network.shuffle.ExternalShuffleBlockHandler.receive(ExternalShuffleBlockHandler.java:68) >> at >> org.apache.spark.network.server.TransportRequestHandler.processRpcRequest(TransportRequestHandler.java:114) >> at >> org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:87) >> at >> org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:101) >> at >> org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:51) >> at >> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308) >> at >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294) >> > [...] > > In the scheduler service I see logs like this on every box: > > log4j:WARN No appenders could be found for logger >> (org.apache.hadoop.metrics2.lib.MutableMetricsFactory). >> log4j:WARN Please initialize the log4j system properly. >> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for >> more info. >> Using Spark's repl log4j profile: >> org/apache/spark/log4j-defaults-repl.properties >> To adjust logging level use sc.setLogLevel("INFO") >> 16/04/14 19:12:29 ERROR TransportRequestHandler: Error while invoking >> RpcHandler#receive() on RPC id 7280403447531815366 >> java.lang.UnsupportedOperationException: Unexpected message: >> org.apache.spark.network.shuffle.protocol.mesos.RegisterDriver@c399c59a >> at >> org.apache.spark.network.shuffle.ExternalShuffleBlockHandler.handleMessage(ExternalShuffleBlockHandler.java:92) >> at >> org.apache.spark.network.shuffle.ExternalShuffleBlockHandler.receive(ExternalShuffleBlockHandler.java:68) >> at >> org.apache.spark.network.server.TransportRequestHandler.processRpcRequest(TransportRequestHandler.java:114) >> at >> org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:87) >> at >> org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:101) >> at >> org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:51) >> at >> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308) >> > [...] > > Thanks! > > - Jo Voordeckers > >
