[
https://issues.apache.org/jira/browse/SPARK-17256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanmao Liu updated SPARK-17256:
--------------------------------
Description:
The key problem is { cmd /V /E /C "~%dp0spark-xxx.cmd" } cannot accept
arguments which has double quotes, even if itself of "spark-xxx.cmd" cannot be
just started.
Double quoted command in spark/bin/spark-submit.cmd (and spark-class.cmd etc)
as following:
cmd /V /E /C "%~dp0spark-submit2.cmd" %*
Will cut off the argument that double quoted, and not effect if full path of
spark-submit.cmd has space.
Example case :
spark-submit.cmd --jars xxxx
"jdbc:mysql://localhost:3306/lzdb?user=guest&password=abc123" my_table
(1) Cannot just start spark-submit.cmd if it's full path has space, for example
"D:\opengit\spark\bin - Copy\spark-submit.cmd" :
D:\opengit\spark\bin - Copy>
spark-submit.cmd --jars just-to-start
"jdbc:mysql://localhost:3306/lzdb?user=guest&password=abc123" my_table
------- will get error :
'D:\opengit\spark\bin' is not recognized as an internal or external command,
operable program or batch file.
'password' is not recognized as an internal or external command,
operable program or batch file.
(2) spark-submit.cmd can be started but will cut off arguments if path's no
space . for example of above but no space:
D:\opengit\spark\bin>
spark-submit.cmd --jars just-to-start
"jdbc:mysql://localhost:3306/lzdb?user=guest&password=abc123" my_table
------- will get error :
'password' is not recognized as an internal or external command,
operable program or batch file.
(1) A solution for this is just removing the "cmd /V /E /C" in the
spark/bin/xxx.cmd files , but this will has possible "pollution" to the
environment variables in the same cmd window (and later derived cmd window like
started in it by "start cmd").
But this will solve the space in path problem.
(2) A mitigation is checking before adding double quotes, and added if really
need. This just works well if path no space.
was:
The key problem is { cmd /V /E /C "~%dp0spark-xxx.cmd" } cannot accept
arguments has double quotes, even if itself of "spark-xxx.cmd".
Double quoted command in spark/bin/spark-submit.cmd (and spark-class.cmd etc)
as following:
cmd /V /E /C "%~dp0spark-submit2.cmd" %*
Will cut off the argument that double quoted, and not effect if full path of
spark-submit.cmd has space.
Example case :
spark-submit.cmd --jars xxxx
"jdbc:mysql://localhost:3306/lzdb?user=guest&password=abc123" my_table
(1) Cannot just start spark-submit.cmd if it's full path has space, for example
"D:\opengit\spark\bin - Copy\spark-submit.cmd" :
D:\opengit\spark\bin - Copy>
spark-submit.cmd --jars just-to-start
"jdbc:mysql://localhost:3306/lzdb?user=guest&password=abc123" my_table
---- will get erorr :
'D:\opengit\spark\bin' is not recognized as an internal or external command,
operable program or batch file.
'password' is not recognized as an internal or external command,
operable program or batch file.
(2) spark-submit.cmd can be started but will cut off arguments if path's no
space . for example of above but no space:
D:\opengit\spark\bin>
spark-submit.cmd --jars just-to-start
"jdbc:mysql://localhost:3306/lzdb?user=guest&password=abc123" my_table
----- will get error :
'password' is not recognized as an internal or external command,
operable program or batch file.
(1) A solution for this is just removing the "cmd /V /E /C" in the
spark/bin/xxx.cmd files , but this will has possible "pollution" to the
environment variables in the same cmd window (and later derived cmd window like
started in it by "start cmd").
But this will solve the space in path problem.
(2) A mitigation is checking before adding double quotes, and added if really
need. This just works well if path no space.
> spark-submit.cmd cannot work if path has space and cut off double-quoted
> arguments
> ----------------------------------------------------------------------------------
>
> Key: SPARK-17256
> URL: https://issues.apache.org/jira/browse/SPARK-17256
> Project: Spark
> Issue Type: Bug
> Components: Deploy
> Affects Versions: 2.0.0
> Environment: Windows
> Reporter: Quanmao Liu
> Labels: cut-off, double-quotes
> Original Estimate: 5m
> Remaining Estimate: 5m
>
> The key problem is { cmd /V /E /C "~%dp0spark-xxx.cmd" } cannot accept
> arguments which has double quotes, even if itself of "spark-xxx.cmd" cannot
> be just started.
> Double quoted command in spark/bin/spark-submit.cmd (and spark-class.cmd etc)
> as following:
> cmd /V /E /C "%~dp0spark-submit2.cmd" %*
> Will cut off the argument that double quoted, and not effect if full path of
> spark-submit.cmd has space.
> Example case :
> spark-submit.cmd --jars xxxx
> "jdbc:mysql://localhost:3306/lzdb?user=guest&password=abc123" my_table
> (1) Cannot just start spark-submit.cmd if it's full path has space, for
> example "D:\opengit\spark\bin - Copy\spark-submit.cmd" :
> D:\opengit\spark\bin - Copy>
> spark-submit.cmd --jars just-to-start
> "jdbc:mysql://localhost:3306/lzdb?user=guest&password=abc123" my_table
> ------- will get error :
> 'D:\opengit\spark\bin' is not recognized as an internal or external command,
> operable program or batch file.
> 'password' is not recognized as an internal or external command,
> operable program or batch file.
> (2) spark-submit.cmd can be started but will cut off arguments if path's no
> space . for example of above but no space:
> D:\opengit\spark\bin>
> spark-submit.cmd --jars just-to-start
> "jdbc:mysql://localhost:3306/lzdb?user=guest&password=abc123" my_table
> ------- will get error :
> 'password' is not recognized as an internal or external command,
> operable program or batch file.
> (1) A solution for this is just removing the "cmd /V /E /C" in the
> spark/bin/xxx.cmd files , but this will has possible "pollution" to the
> environment variables in the same cmd window (and later derived cmd window
> like started in it by "start cmd").
> But this will solve the space in path problem.
> (2) A mitigation is checking before adding double quotes, and added if really
> need. This just works well if path no space.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
