[GitHub] spark issue #19090: [SPARK-21877][DEPLOY, WINDOWS] Handle quotes in Windows ...
Github user minixalpha commented on the issue: https://github.com/apache/spark/pull/19090 Thanks, @HyukjinKwon @jsnowacki @felixcheung --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19090: [SPARK-21877][DEPLOY, WINDOWS] Handle quotes in Windows ...
Github user minixalpha commented on the issue: https://github.com/apache/spark/pull/19090 @jsnowacki I have already add comments to explain the quotes, could you help me review the comments? Thanks. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19090: [SPARK-21877][DEPLOY, WINDOWS] Handle quotes in Windows ...
Github user minixalpha commented on the issue: https://github.com/apache/spark/pull/19090 @jsnowacki Thanks for reviewing this PR! There are some situations people cannot omit the quotes, such as multiple parameters of "--driver-java-options". For example: [passing multiple -D arguments to driver-java-options in spark-submit on windows](https://stackoverflow.com/questions/44578225/passing-multiple-d-arguments-to-driver-java-options-in-spark-submit-on-windows/45972447) Actually, I find this bug when I try to start a Spark interpreter in Apache Zeppelin. When SPARK_HOME is set, Zeppelin will add some options to spark-submit, in these options, there are some quotes, which trigger this bug in Windows. I trace the launch process of Spark interpreter and finally I found that it is a bug of Spark. Without this bugfix, Zeppelin cannot start Spark interpreter when SPARK_HOME is set on Windows. I will add some comments about these quotes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #9183: [SPARK-11215] [ML] Add multiple columns support to String...
Github user minixalpha commented on the issue: https://github.com/apache/spark/pull/9183 Thanks for you job! @WeichenXu123 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #9183: [SPARK-11215] [ML] Add multiple columns support to String...
Github user minixalpha commented on the issue: https://github.com/apache/spark/pull/9183 @WeichenXu123 Any activity for the new PR? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19090: [SPARK-21877][DEPLOY, WINDOWS] Handle quotes in Windows ...
Github user minixalpha commented on the issue: https://github.com/apache/spark/pull/19090 I design two groups test cases: - Test cases about windows command scripts options - Examples in Spark Document All these test cases works well. ## Test cases about windows command scripts options All these test cases take `bin\spark-shell` as example, as other commands works similarly. For each test case, I record all the java program options when run class `org.apache.spark.launcher.Main` and `org.apache.spark.deploy.SparkSubmit`, and check the options. ### No option ``` bin\spark-shell "C:\jdk1.8.0_65\bin\java" -Xmx128m -cp ""C:\spark-pr-19090\spark-2.2.0-bin-hadoop2.7-fix\bin\..\jars"\*" org.apache.spark.launcher.Main org.apache.spark.deploy.SparkSubmit --class org.apache.spark.repl.Main --name "Spark shell" C:\jdk1.8.0_65\bin\java -cp "C:\spark-pr-19090\spark-2.2.0-bin-hadoop2.7-fix\bin\..\conf\;C:\spark-pr-19090\spark-2.2.0-bin-hadoop2.7\bin\..\jars\*" "-Dscala.usejavacp=true" -Xmx1g org.apache.spark.deploy.SparkSubmit --class org.apache.spark.repl.Main --name "Spark shell" spark-shell ``` ### Has options One option # Option has no parameter ``` bin\spark-shell --verbose "C:\jdk1.8.0_65\bin\java" -Xmx128m -cp ""c:\spark-pr-19090\spark-2.2.0-bin-hadoop2.7-fix\bin\..\jars"\*" org.apache.spark.launcher.Main org.apache.spark.deploy.SparkSubmit --class org.apache.spark.repl.Main --name "Spark shell" --verbose C:\jdk1.8.0_65\bin\java -cp "c:\spark-pr-19090\spark-2.2.0-bin-hadoop2.7-fix\bin\..\conf\;c:\spark-pr-19090\spark-2.2.0-bin-hadoop2.7-fix\bin\..\jars\*" "-Dscala.usejavacp=true" -Xmx1g org.apache.spark.deploy.SparkSubmit --class org.apache.spark.repl.Main --name "Spark shell" --verbose spark-shell ``` # Option has parameter ## Option parameter has no quotes ``` bin\spark-shell --driver-java-options -Dfile.encoding=utf-8 "C:\jdk1.8.0_65\bin\java" -Xmx128m -cp ""c:\spark-pr-19090\spark-2.2.0-bin-hadoop2.7-fix\bin\..\jars"\*" org.apache.spark.launcher.Main org.apache.spark.deploy.SparkSubmit --class org.apache.spark.repl.Main --name "Spark shell" --driver-java-options -Dfile.encoding=utf-8 C:\jdk1.8.0_65\bin\java -cp "c:\spark-pr-19090\spark-2.2.0-bin-hadoop2.7-fix\bin\..\conf\;c:\spark-pr-19090\spark-2.2.0-bin-hadoop2.7-fix\bin\..\jars\*" "-Dscala.usejavacp=true" -Xmx1g "-Dfile.encoding=utf-8" org.apache.spark.deploy.SparkSubmit --conf "spark.driver.extraJavaOptions=-Dfile.encoding=utf-8" --class org.apache.spark.repl.Main --name "Spark shell" spark-shell ``` ## Option parameter has quotes - quotes one parameter ``` bin\spark-shell --driver-java-options "-Dfile.encoding=utf-8" "C:\jdk1.8.0_65\bin\java" -Xmx128m -cp ""c:\spark-pr-19090\spark-2.2.0-bin-hadoop2.7-fix\bin\..\jars"\*" org.apache.spark.launcher.Main org.apache.spark.deploy.SparkSubmit --class org.apache.spark.repl.Main --name "Spark shell" --driver-java-options "-Dfile.encoding=utf-8" C:\jdk1.8.0_65\bin\java -cp "c:\spark-pr-19090\spark-2.2.0-bin-hadoop2.7-fix\bin\..\conf\;c:\spark-pr-19090\spark-2.2.0-bin-hadoop2.7-fix\bin\..\jars\*" "-Dscala.usejavacp=true" -Xmx1g "-Dfile.encoding=utf-8" org.apache.spark.deploy.SparkSubmit --conf "spark.driver.extraJavaOptions=-Dfile.encoding=utf-8" --class org.apache.spark.repl.Main --name "Spark shell" spark-shell ``` - quotes multi parameter ``` bin\spark-shell --driver-java-options "-Dfile.encoding=utf-8 -Dsun.jnu.encoding=utf-8" "C:\jdk1.8.0_65\bin\java" -Xmx128m -cp ""c:\spark-pr-19090\spark-2.2.0-bin-hadoop2.7-fix\bin\..\jars"\*" org.apache.spark.launcher.Main org.apache.spark.deploy.SparkSubmit --class org.apache.spark.repl.Main --name "Spark shell" --driver-java-options "-Dfile.encoding=utf-8 -Dsun.jnu.encoding=utf-8" C:\jdk1.8.0_65\bin\java -cp "c:\spark-pr-19090\spark-2.2.0-bin-hadoop2.7-fix\bin\..\conf\;c:\spark-pr-19090\spark-2.2.0-bin-hadoop2.7-fix\bin\..\jars\*" "-Dscala.usejavacp=true" -Xmx1g "-Dfile.encoding=utf-8" "-Dsun.jnu.encoding=utf-8" org.apache.spark.deploy.SparkSubmit --conf "spark.driver.extraJavaOptions=-Dfile.encoding=utf-8 -Dsun.jnu.encoding=utf-8" --class org.apache.spark.repl.Main --name "Spark shell" spark-shell ``` Multi options # all op
[GitHub] spark issue #19090: [SPARK-21877][DEPLOY, WINDOWS] Handle quotes in Windows ...
Github user minixalpha commented on the issue: https://github.com/apache/spark/pull/19090 ok, I will give more test cases later. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19090: [SPARK-21877][DEPLOY, WINDOWS] Handle quotes in Windows ...
Github user minixalpha commented on the issue: https://github.com/apache/spark/pull/19090 @HyukjinKwon Thanks for your review! Should I provide more test cases to cover the potential corner cases? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19090: [SPARK-21877][DEPLOY, WINDOWS] Handle quotes in W...
GitHub user minixalpha opened a pull request: https://github.com/apache/spark/pull/19090 [SPARK-21877][DEPLOY, WINDOWS] Handle quotes in Windows command scripts ## What changes were proposed in this pull request? All the windows command scripts can not handle quotes in parameter. Run a windows command shell with parameter which has quotes can reproduce the bug: ``` C:\Users\meng\software\spark-2.2.0-bin-hadoop2.7> bin\spark-shell --driver-java-options " -Dfile.encoding=utf-8 " 'C:\Users\meng\software\spark-2.2.0-bin-hadoop2.7\bin\spark-shell2.cmd" --driver-java-options "' is not recognized as an internal or external command, operable program or batch file. ``` Windows recognize "--driver-java-options" as part of the command. All the Windows command script has the following code have the bug. ``` cmd /V /E /C "" %* ``` We should quote command and parameters like ``` cmd /V /E /C """ %*" ``` We can verify it by the following demo: ``` C:\Users\meng\program\demo>cat a.cmd @echo off cmd /V /E /C "b.cmd" %* C:\Users\meng\program\demo>cat b.cmd @echo off echo %* C:\Users\meng\program\demo>cat c.cmd @echo off cmd /V /E /C ""b.cmd" %*" C:\Users\meng\program\demo>a.cmd "123" 'b.cmd" "123' is not recognized as an internal or external command, operable program or batch file. C:\Users\meng\program\demo>c.cmd "123" "123" ``` ## How was this patch tested? Test manually on Windows 10. (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) Please review http://spark.apache.org/contributing.html before opening a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/minixalpha/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/19090.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #19090 commit 26fc75670750b4710a2e0d97642ec1b52d51ffbb Author: minixalpha Date: 2017-08-31T00:31:39Z handle quotes in Windows command scripts --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
