Quanmao Liu created SPARK-17256:
-----------------------------------
Summary: spark-submit.cmd cannot work if path has space and cut
off double-quoted arguments
Key: SPARK-17256
URL: https://issues.apache.org/jira/browse/SPARK-17256
Project: Spark
Issue Type: Bug
Components: Deploy
Affects Versions: 2.0.0
Environment: Windows
Reporter: Quanmao Liu
The key problem is { cmd /V /E /C "~%dp0spark-xxx.cmd" } cannot accept
arguments has double quotes, even if itself of "spark-xxx.cmd".
Double quoted command in spark/bin/spark-submit.cmd (and spark-class.cmd etc)
as following:
cmd /V /E /C "%~dp0spark-submit2.cmd" %*
Will cut off the argument that double quoted, and not effect if full path of
spark-submit.cmd has space.
Example case :
spark-submit.cmd --jars xxxx
"jdbc:mysql://localhost:3306/lzdb?user=guest&password=abc123" my_table
(1) Cannot just start spark-submit.cmd if it's full path has space, for example
"D:\opengit\spark\bin - Copy\spark-submit.cmd" :
D:\opengit\spark\bin - Copy>
spark-submit.cmd --jars just-to-start
"jdbc:mysql://localhost:3306/lzdb?user=guest&password=abc123" my_table
---- will get erorr :
'D:\opengit\spark\bin' is not recognized as an internal or external command,
operable program or batch file.
'password' is not recognized as an internal or external command,
operable program or batch file.
(2) spark-submit.cmd can be started but will cut off arguments if path's no
space . for example of above but no space:
D:\opengit\spark\bin>
spark-submit.cmd --jars just-to-start
"jdbc:mysql://localhost:3306/lzdb?user=guest&password=abc123" my_table
----- will get error :
'password' is not recognized as an internal or external command,
operable program or batch file.
(1) A solution for this is just removing the "cmd /V /E /C" in the
spark/bin/xxx.cmd files , but this will has possible "pollution" to the
environment variables in the same cmd window (and later derived cmd window like
started in it by "start cmd").
But this will solve the space in path problem.
(2) A mitigation is checking before adding double quotes, and added if really
need. This just works well if path no space.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
