[
https://issues.apache.org/jira/browse/SPARK-12800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen updated SPARK-12800:
------------------------------
Description:
Version used: Spark 1.5.1 (1.5.2-SNAPSHOT)
Deployment Mode: Yarn-Cluster
Problem observed:
When running spark job directly from YarnClient (without using spark-submit,
I did not verify the spark-submit has the same issue or not), when kerberos
security is enabled, the first time run spark job always fail. The failure is
due to that the hadoop consider the job is in SIMPLE model rather than Kerberos
mode. But without shutting down the JVM, run the same job again, the spark job
will pass. If one restart the JVM, then the spark job will fail again.
The cause:
Tracking down the source of the issue, I found that the problem seems lie at
the spark Yarn Client.scala. In the Client
{code}
def prepareLocalResources() method L 266 of Client.java, the following line
code is called.
YarnSparkHadoopUtil.get.obtainTokensForNamenodes(nns, hadoopConf, credentials)
{code}
The YarnSparkHadoopUtil.get is in turns get initialized via reflection
{code}
object SparkHadoopUtil {
private val hadoop = {
val yarnMode = java.lang.Boolean.valueOf(
System.getProperty("SPARK_YARN_MODE", System.getenv("SPARK_YARN_MODE")))
if (yarnMode) {
try {
Utils.classForName("org.apache.spark.deploy.yarn.YarnSparkHadoopUtil")
.newInstance()
.asInstanceOf[SparkHadoopUtil]
} catch {
case e: Exception => throw new SparkException("Unable to load YARN
support", e)
}
} else {
new SparkHadoopUtil
}
}
def get: SparkHadoopUtil = {
hadoop
}
}
class SparkHadoopUtil extends Logging {
private val sparkConf = new SparkConf()
val conf: Configuration = newConfiguration(sparkConf)
UserGroupInformation.setConfiguration(conf)
.... rest of line
}
{code}
Here SparkHadoopUtil creates a empty SparkConf and Hadoop Configuration from
that and set to UserGroupInformation
{code}
UserGroupInformation.setConfiguration(conf)
{code}
As the UserGroupInformation.authenticationMethod is static, above all wipe
out the security settings. UserGroupInformation.isSecurityEnabled() changed
from true to false. Thus the sequence call will fail.
Since the SparkHadoopUtil.hadoop is static/non-mutable variable, so
the next run it will be not create again, then
UserGroupInformation.setConfiguration(conf)
will not be called again, so the sequence spark job works.
The work around:
{code}
//first initialize the SparkHadoopUtil, which will create a static
instance
//which will set UserGroupInformation to a empty hadoop Configuration.
//we will need to reset the UserGroupInformation after that.
val util = SparkHadoopUtil.get
UserGroupInformation.setConfiguration(hadoopConf)
{code}
Then call
client.run()
was:
Version used: Spark 1.5.1 (1.5.2-SNAPSHOT)
Deployment Mode: Yarn-Cluster
Problem observed:
When running spark job directly from YarnClient (without using spark-submit,
I did not verify the spark-submit has the same issue or not), when kerberos
security is enabled, the first time run spark job always fail. The failure is
due to that the hadoop consider the job is in SIMPLE model rather than Kerberos
mode. But without shutting down the JVM, run the same job again, the spark job
will pass. If one restart the JVM, then the spark job will fail again.
The cause:
Tracking down the source of the issue, I found that the problem seems lie at
the spark Yarn Client.scala. In the Client
def prepareLocalResources() method L 266 of Client.java, the following line
code is called.
YarnSparkHadoopUtil.get.obtainTokensForNamenodes(nns, hadoopConf, credentials)
The YarnSparkHadoopUtil.get is in turns get initialized via reflection
object SparkHadoopUtil {
private val hadoop = {
val yarnMode = java.lang.Boolean.valueOf(
System.getProperty("SPARK_YARN_MODE", System.getenv("SPARK_YARN_MODE")))
if (yarnMode) {
try {
Utils.classForName("org.apache.spark.deploy.yarn.YarnSparkHadoopUtil")
.newInstance()
.asInstanceOf[SparkHadoopUtil]
} catch {
case e: Exception => throw new SparkException("Unable to load YARN
support", e)
}
} else {
new SparkHadoopUtil
}
}
def get: SparkHadoopUtil = {
hadoop
}
}
class SparkHadoopUtil extends Logging {
private val sparkConf = new SparkConf()
val conf: Configuration = newConfiguration(sparkConf)
UserGroupInformation.setConfiguration(conf)
.... rest of line
}
Here SparkHadoopUtil creates a empty SparkConf and Hadoop Configuration from
that and set to UserGroupInformation
UserGroupInformation.setConfiguration(conf)
As the UserGroupInformation.authenticationMethod is static, above all wipe
out the security settings. UserGroupInformation.isSecurityEnabled() changed
from true to false. Thus the sequence call will fail.
Since the SparkHadoopUtil.hadoop is static/non-mutable variable, so
the next run it will be not create again, then
UserGroupInformation.setConfiguration(conf)
will not be called again, so the sequence spark job works.
The work around:
//first initialize the SparkHadoopUtil, which will create a static
instance
//which will set UserGroupInformation to a empty hadoop Configuration.
//we will need to reset the UserGroupInformation after that.
val util = SparkHadoopUtil.get
UserGroupInformation.setConfiguration(hadoopConf)
Then call
client.run()
> Subtle bug on Spark Yarn Client under Kerberos Security Mode
> ------------------------------------------------------------
>
> Key: SPARK-12800
> URL: https://issues.apache.org/jira/browse/SPARK-12800
> Project: Spark
> Issue Type: Bug
> Affects Versions: 1.5.1, 1.5.2
> Reporter: Chester
>
> Version used: Spark 1.5.1 (1.5.2-SNAPSHOT)
> Deployment Mode: Yarn-Cluster
> Problem observed:
> When running spark job directly from YarnClient (without using
> spark-submit, I did not verify the spark-submit has the same issue or not),
> when kerberos security is enabled, the first time run spark job always fail.
> The failure is due to that the hadoop consider the job is in SIMPLE model
> rather than Kerberos mode. But without shutting down the JVM, run the same
> job again, the spark job will pass. If one restart the JVM, then the spark
> job will fail again.
> The cause:
> Tracking down the source of the issue, I found that the problem seems lie
> at the spark Yarn Client.scala. In the Client
> {code}
> def prepareLocalResources() method L 266 of Client.java, the following line
> code is called.
> YarnSparkHadoopUtil.get.obtainTokensForNamenodes(nns, hadoopConf,
> credentials)
> {code}
> The YarnSparkHadoopUtil.get is in turns get initialized via reflection
> {code}
> object SparkHadoopUtil {
> private val hadoop = {
> val yarnMode = java.lang.Boolean.valueOf(
> System.getProperty("SPARK_YARN_MODE",
> System.getenv("SPARK_YARN_MODE")))
> if (yarnMode) {
> try {
> Utils.classForName("org.apache.spark.deploy.yarn.YarnSparkHadoopUtil")
> .newInstance()
> .asInstanceOf[SparkHadoopUtil]
> } catch {
> case e: Exception => throw new SparkException("Unable to load YARN
> support", e)
> }
> } else {
> new SparkHadoopUtil
> }
> }
> def get: SparkHadoopUtil = {
> hadoop
> }
> }
>
> class SparkHadoopUtil extends Logging {
> private val sparkConf = new SparkConf()
> val conf: Configuration = newConfiguration(sparkConf)
> UserGroupInformation.setConfiguration(conf)
> .... rest of line
> }
> {code}
> Here SparkHadoopUtil creates a empty SparkConf and Hadoop Configuration from
> that and set to UserGroupInformation
> {code}
> UserGroupInformation.setConfiguration(conf)
> {code}
> As the UserGroupInformation.authenticationMethod is static, above all wipe
> out the security settings. UserGroupInformation.isSecurityEnabled() changed
> from true to false. Thus the sequence call will fail.
> Since the SparkHadoopUtil.hadoop is static/non-mutable variable, so
> the next run it will be not create again, then
> UserGroupInformation.setConfiguration(conf)
> will not be called again, so the sequence spark job works.
> The work around:
> {code}
> //first initialize the SparkHadoopUtil, which will create a static
> instance
> //which will set UserGroupInformation to a empty hadoop Configuration.
> //we will need to reset the UserGroupInformation after that.
> val util = SparkHadoopUtil.get
> UserGroupInformation.setConfiguration(hadoopConf)
> {code}
> Then call
> client.run()
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
