[
https://issues.apache.org/jira/browse/SPARK-8302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Imran Rashid resolved SPARK-8302.
---------------------------------
Resolution: Fixed
Fix Version/s: 1.5.0
Issue resolved by pull request 6752
[https://github.com/apache/spark/pull/6752]
> Support heterogeneous cluster nodes on YARN
> -------------------------------------------
>
> Key: SPARK-8302
> URL: https://issues.apache.org/jira/browse/SPARK-8302
> Project: Spark
> Issue Type: New Feature
> Components: YARN
> Affects Versions: 1.5.0
> Reporter: Marcelo Vanzin
> Fix For: 1.5.0
>
>
> Some of our customers install Hadoop on different paths across the cluster.
> When running a Spark app, this leads to a few complications because of how we
> try to reuse the rest of Hadoop.
> Since all configuration for a Spark-on-YARN application is local, the code
> does not have enough information about how to run things on the rest of the
> cluster in such cases.
> To illustrate: let's say that a node's configuration says that
> {{SPARK_DIST_CLASSPATH=/disk1/hadoop/lib/*}}. If I launch a Spark app from
> that machine, but there's a machine on the cluster where Hadoop is actually
> installed in {{/disk2/hadoop/lib}}, then any container launched on that node
> will fail.
> The problem does not exist (or is much less pronounced) on standalone and
> mesos since they require a local Spark installation and configuration.
> It would be nice if we could easily support this use case on YARN.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
