[
https://issues.apache.org/jira/browse/SPARK-21494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcelo Vanzin resolved SPARK-21494.
------------------------------------
Resolution: Fixed
Assignee: Marcelo Vanzin
Fix Version/s: 2.3.0
2.2.1
> Spark 2.2.0 AES encryption not working with External shuffle
> ------------------------------------------------------------
>
> Key: SPARK-21494
> URL: https://issues.apache.org/jira/browse/SPARK-21494
> Project: Spark
> Issue Type: Bug
> Components: Block Manager, Shuffle
> Affects Versions: 2.2.0
> Environment: AWS EMR
> Reporter: Udit Mehrotra
> Assignee: Marcelo Vanzin
> Fix For: 2.2.1, 2.3.0
>
> Attachments: logs.zip
>
>
> Spark’s new AES based authentication mechanism does not seem to work when
> configured with external shuffle service on YARN.
> Here is the stack trace for the error we see in the driver logs:
> ERROR YarnScheduler: Lost executor 40 on ip-10-167-104-125.ec2.internal:
> Unable to create executor due to Unable to register with external shuffle
> server due to: java.lang.IllegalArgumentException: Authentication failed.
> at
> org.apache.spark.network.crypto.AuthRpcHandler.receive(AuthRpcHandler.java:125)
> at
> org.apache.spark.network.server.TransportRequestHandler.processRpcRequest(TransportRequestHandler.java:157)
> at
> org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:105)
> at
> org.apache.spark.network.server.TransportChannelHandler.channelRead(TransportChannelHandler.java:118)
> at
> org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
> at
> org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
> at
> org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336)
> at
> org.spark_project.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
> at
> org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
> at
> org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
> at
> org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336)
> at
> org.spark_project.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
>
> Here are the settings we are configuring in ‘spark-defaults’ and ‘yarn-site’:
> spark.network.crypto.enabled true
> spark.network.crypto.saslFallback false
> spark.authenticate true
>
> Turning on DEBUG logs for class ‘org.apache.spark.network.crypto’ on both
> Spark and YARN side is not giving much information either about why
> authentication fails. The driver and node manager logs have been attached to
> the JIRA.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
