[
https://issues.apache.org/jira/browse/SPARK-35124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17325228#comment-17325228
]
Malthe Borch commented on SPARK-35124:
--------------------------------------
The issue seems to stem from the way Python's "Popen" command handles arguments
on Windows – basically it doesn't have a clue that "&" has a special meaning on
Windows (and a number of other characters – in particular {{&}}, {{|}}, {{(}},
{{)}}, {{<}}, {{>}}, {{^}}).
> Local mode fails to start cluster due to configuration value escape issue
> -------------------------------------------------------------------------
>
> Key: SPARK-35124
> URL: https://issues.apache.org/jira/browse/SPARK-35124
> Project: Spark
> Issue Type: Bug
> Components: Java API
> Affects Versions: 3.1.1
> Reporter: Malthe Borch
> Priority: Major
>
> At least on Windows and perhaps also other systems, running Spark in local
> mode fails to start a cluster when a configuration key contains a value with
> "&" (ampersand) in it.
> This happens during the "getOrCreate()" call from a Spark session builder.
> The reason seems to be incorrect or insufficient escaping since on Windows,
> the attempt to start a Spark process actually ends up running multiple
> commands for each occurrence of the "&" character.
> On Windows specifically, the correct way to escape "&" would be "^&", but I
> have not been able yet to figure out exactly where the process is started and
> how the configuration is passed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
