Hmm... that's not enough info and logs are intentionally kept silent to avoid flooding, but if you enable DEBUG level logging for org.apache.spark.network.crypto in both YARN and the Spark app, that might provide more info.
On Wed, Jul 19, 2017 at 2:58 PM, Udit Mehrotra <udit.mehrotr...@gmail.com> wrote: > So I added these settings in yarn-site.xml as well. Now I get a completely > different error, but atleast it seems like it is using the crypto library: > > ExecutorLostFailure (executor 1 exited caused by one of the running tasks) > Reason: Unable to create executor due to Unable to register with external > shuffle server due to : java.lang.IllegalArgumentException: Authentication > failed. > at > org.apache.spark.network.crypto.AuthRpcHandler.receive(AuthRpcHandler.java:125) > at > org.apache.spark.network.server.TransportRequestHandler.processRpcRequest(TransportRequestHandler.java:157) > at > org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:105) > at > org.apache.spark.network.server.TransportChannelHandler.channelRead(TransportChannelHandler.java:118) > > Any clue about this ? > > > On Wed, Jul 19, 2017 at 1:13 PM, Marcelo Vanzin <van...@cloudera.com> wrote: >> >> On Wed, Jul 19, 2017 at 1:10 PM, Udit Mehrotra >> <udit.mehrotr...@gmail.com> wrote: >> > Is there any additional configuration I need for external shuffle >> > besides >> > setting the following: >> > spark.network.crypto.enabled true >> > spark.network.crypto.saslFallback false >> > spark.authenticate true >> >> Have you set these options on the shuffle service configuration too >> (which is the YARN xml config file, not spark-defaults.conf)? >> >> If you have there might be an issue, and you should probably file a >> bug and include your NM's log file. >> >> -- >> Marcelo > > -- Marcelo --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
