[
https://issues.apache.org/jira/browse/SPARK-8302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-8302:
-----------------------------------
Assignee: (was: Apache Spark)
> Support heterogeneous cluster nodes on YARN
> -------------------------------------------
>
> Key: SPARK-8302
> URL: https://issues.apache.org/jira/browse/SPARK-8302
> Project: Spark
> Issue Type: New Feature
> Components: YARN
> Affects Versions: 1.5.0
> Reporter: Marcelo Vanzin
>
> Some of our customers install Hadoop on different paths across the cluster.
> When running a Spark app, this leads to a few complications because of how we
> try to reuse the rest of Hadoop.
> Since all configuration for a Spark-on-YARN application is local, the code
> does not have enough information about how to run things on the rest of the
> cluster in such cases.
> To illustrate: let's say that a node's configuration says that
> {{SPARK_DIST_CLASSPATH=/disk1/hadoop/lib/*}}. If I launch a Spark app from
> that machine, but there's a machine on the cluster where Hadoop is actually
> installed in {{/disk2/hadoop/lib}}, then any container launched on that node
> will fail.
> The problem does not exist (or is much less pronounced) on standalone and
> mesos since they require a local Spark installation and configuration.
> It would be nice if we could easily support this use case on YARN.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
