[
https://issues.apache.org/jira/browse/SPARK-7807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14555312#comment-14555312
]
Norman He commented on SPARK-7807:
----------------------------------
Running spark in local mode or mesos cluster, The hadoopConfiguration need to
know all the core-site.xml and hdfs-site.xml from some http://url service. Due
to bundle issue (there are a lot of core-site.xml and hdfs-site.xml in all
kinds of testing jar), the spark instantiate hadoopConfiguration won't be able
to pick up the correct resource in HDFS high availabity setup.
Add the spark.hadoop.url support can be one way to solve this issue and the
clean way.
> High-Availablity:: SparkHadoopUtil.scala should support
> hadoopConfiguration.addResource()
> ------------------------------------------------------------------------------------------
>
> Key: SPARK-7807
> URL: https://issues.apache.org/jira/browse/SPARK-7807
> Project: Spark
> Issue Type: Improvement
> Environment: running spark against remote-hadoop HA cluster. Easy of
> use with spark.hadoop.url. prefix.
> 1) user can support sparkConf with prefix spark.hadoop.url. like
> spark.hadoop.url.core-site
> and spark.hadoop.url.hdfs-site
> Reporter: Norman He
> Priority: Trivial
> Labels: easyfix
>
> line 97 : should below should be able to change to
> conf.getAll.foreach { case (key, value) =>
> if (key.startsWith("spark.hadoop.")) {
> hadoopConf.set(key.substring("spark.hadoop.".length), value)
> }
> }
> ----------------new version-------------------------------
> conf.getAll.foreach { case (key, value) =>
> if (key.startsWith("spark.hadoop.")) {
> if( key.startsWith("spark.hadoop.url."))
> hadoopConf.addResource(new URL(value))
> else
> hadoopConf.set(key.substring("spark.hadoop.".length), value)
> }
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
