[
https://issues.apache.org/jira/browse/SPARK-12176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuhang Chen updated SPARK-12176:
--------------------------------
Description:
The spark-submit uses '--conf K=V' pattern for setting configs. According to
the docs, if the 'V' you set has spaces in it, the whole 'K=V' parts should be
wrapped with quotes.
However, the SparkLauncher (org.apache.spark.launcher.SparkLauncher) would not
do that wrapping for you, and there is no chance for wrapping by yourself with
the API it provides.
I checked up the source, all confs are stored in a Map before generating
launching commands. Thus. my advice is checking all values of the conf Map and
do wrapping during command building.
was:
The spark-submit uses '--conf K=V' pattern for setting configs. And according
to the docs, if the 'V' you set contains spaces in it, you should wrap the
whole 'K=V' parts with quotes. However, the SparkLauncher
(org.apache.spark.launcher.SparkLauncher) would not wrap the ‘K=V' parts for
you, and there is no place for wrapping by yourself with the API it provides.
I checked up the source, all confs are stored in a Map before generating
launching commands. Thus. my advice is checking all values of the conf Map and
do wrapping during command building.
> SparkLauncher's setConf() does not support configs containing spaces
> --------------------------------------------------------------------
>
> Key: SPARK-12176
> URL: https://issues.apache.org/jira/browse/SPARK-12176
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.4.0, 1.4.1, 1.5.0, 1.5.1, 1.5.2
> Environment: All
> Reporter: Yuhang Chen
> Priority: Minor
>
> The spark-submit uses '--conf K=V' pattern for setting configs. According to
> the docs, if the 'V' you set has spaces in it, the whole 'K=V' parts should
> be wrapped with quotes.
> However, the SparkLauncher (org.apache.spark.launcher.SparkLauncher) would
> not do that wrapping for you, and there is no chance for wrapping by yourself
> with the API it provides.
> I checked up the source, all confs are stored in a Map before generating
> launching commands. Thus. my advice is checking all values of the conf Map
> and do wrapping during command building.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
