[
https://issues.apache.org/jira/browse/SPARK-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Pivovarov updated SPARK-13532:
----------------------------------------
Description:
Spark yarn executor container fails if yarn.nodemanager.local-dirs starts with
file://
{code}
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>file:///data01/yarn/nm,file:///data02/yarn/nm</value>
</property>
{code}
other application, e.g. Hadoop MR and Hive work normally
Spark works only if yarn.nodemanager.local-dirs does not have file:// prefix
e.g.
{code}
<value>/data01/yarn/nm,/data02/yarn/nm</value>
{code}
to reproduce the issue
open spark-shell
run
{code}
$ spark-shell
> sc.parallelize(1 to 10).count
{code}
was:
Spark yarn executor container fails if yarn.nodemanager.local-dirs starts with
file://
{code}
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>file:///data01/yarn/nm,file:///data02/yarn/nm</value>
</property>
{code}
other application, e.g. Hadoop MR and Hive work normally
Spark works only if yarn.nodemanager.local-dirs does not have file:// prefix
e.g.
{code}
<value>/data01/yarn/nm,/data02/yarn/nm</value>
{code}
> Spark yarn executor container fails if yarn.nodemanager.local-dirs starts
> with file://
> --------------------------------------------------------------------------------------
>
> Key: SPARK-13532
> URL: https://issues.apache.org/jira/browse/SPARK-13532
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 1.6.0
> Reporter: Alexander Pivovarov
>
> Spark yarn executor container fails if yarn.nodemanager.local-dirs starts
> with file://
> {code}
> <property>
> <name>yarn.nodemanager.local-dirs</name>
> <value>file:///data01/yarn/nm,file:///data02/yarn/nm</value>
> </property>
> {code}
> other application, e.g. Hadoop MR and Hive work normally
> Spark works only if yarn.nodemanager.local-dirs does not have file:// prefix
> e.g.
> {code}
> <value>/data01/yarn/nm,/data02/yarn/nm</value>
> {code}
> to reproduce the issue
> open spark-shell
> run
> {code}
> $ spark-shell
> > sc.parallelize(1 to 10).count
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
