[
https://issues.apache.org/jira/browse/SPARK-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Graves updated SPARK-1384:
---------------------------------
Description:
I've found an issue with the spark-shell in yarn-client mode in the 0.9.1 rc3
release. It doesn't work with secure HDFS unless you
export SPARK_YARN_MODE=true before starting the shell, or if you happen to do
something immediately with HDFS. If you wait for the connection to the
namenode to timeout it will fail.
The fix actually went in to master branch with the authentication changes I
made in master but I never realized that change needed to apply to 0.9.
https://github.com/apache/spark/commit/7edbea41b43e0dc11a2de156be220db8b7952d01#diff-0ae5b834ce90ec37c19af35aa7a5e1a0
See the SparkILoop diff.
was:
I've found an issue with the spark-shell in yarn-client mode in the 0.9.1 rc3
release. It doesn't work with secure HDFS unless you
export SPARK_YARN_MODE=true before starting the shell, or if you happen to do
something immediately with HDFS. If you wait for the connection to the
namenode to timeout it will fail.
I think it was actually this way in the 0.9 release also so I thought I would
send this and get peoples feedback to see if you want it fixed?
Another option would be to document that you have to export
SPARK_YARN_MODE=true for the shell. The fix actually went in with the
authentication changes I made in master but I never realized that change needed
to apply to 0.9.
https://github.com/apache/spark/commit/7edbea41b43e0dc11a2de156be220db8b7952d01#diff-0ae5b834ce90ec37c19af35aa7a5e1a0
See the SparkILoop diff.
> spark-shell on yarn on spark 0.9 branch doesn't always work with secure hdfs
> ----------------------------------------------------------------------------
>
> Key: SPARK-1384
> URL: https://issues.apache.org/jira/browse/SPARK-1384
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 0.9.0, 0.9.1
> Reporter: Thomas Graves
>
> I've found an issue with the spark-shell in yarn-client mode in the 0.9.1
> rc3 release. It doesn't work with secure HDFS unless you
> export SPARK_YARN_MODE=true before starting the shell, or if you happen to do
> something immediately with HDFS. If you wait for the connection to the
> namenode to timeout it will fail.
>
> The fix actually went in to master branch with the authentication changes I
> made in master but I never realized that change needed to apply to 0.9.
> https://github.com/apache/spark/commit/7edbea41b43e0dc11a2de156be220db8b7952d01#diff-0ae5b834ce90ec37c19af35aa7a5e1a0
> See the SparkILoop diff.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
