[
https://issues.apache.org/jira/browse/SPARK-7807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14555769#comment-14555769
]
Sean Owen commented on SPARK-7807:
----------------------------------
I don't think we should build a brand new mechanism for this. You can always
add the resources in your program. You're trying to enable a pretty
non-standard configuration -- running on Hadoop but using Mesos?
> High-Availablity:: SparkHadoopUtil.scala should support
> hadoopConfiguration.addResource()
> ------------------------------------------------------------------------------------------
>
> Key: SPARK-7807
> URL: https://issues.apache.org/jira/browse/SPARK-7807
> Project: Spark
> Issue Type: Improvement
> Environment: running spark against remote-hadoop HA cluster. Easy of
> use with spark.hadoop.url. prefix.
> 1) user can support sparkConf with prefix spark.hadoop.url. like
> spark.hadoop.url.core-site
> and spark.hadoop.url.hdfs-site
> Reporter: Norman He
> Priority: Trivial
> Labels: easyfix
>
> line 97 : should below should be able to change to
> conf.getAll.foreach { case (key, value) =>
> if (key.startsWith("spark.hadoop.")) {
> hadoopConf.set(key.substring("spark.hadoop.".length), value)
> }
> }
> ----------------new version-------------------------------
> conf.getAll.foreach { case (key, value) =>
> if (key.startsWith("spark.hadoop.")) {
> if( key.startsWith("spark.hadoop.url."))
> hadoopConf.addResource(new URL(value))
> else
> hadoopConf.set(key.substring("spark.hadoop.".length), value)
> }
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
