[jira] [Updated] (SPARK-23464) MesosClusterScheduler double-escapes parameters to bash command
[ https://issues.apache.org/jira/browse/SPARK-23464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcin Kurczych updated SPARK-23464: Description: Parameters passed to driver launching command in Mesos container are escaped using _shellEscape_ function. In SPARK-18114 additional wrapping in double quotes has been introduced. This cancels out quoting done by _shellEscape_ and makes in unable to run tasks with whitespaces in parameters, as they are interpreted as additional parameters to in-container spark-submit. This is how parameter passed to in-container spark-submit looks like now: {code:java} --conf "spark.executor.extraJavaOptions="-Dfoo=\"first value\" -Dbar=another"" {code} This is how they look after reverting SPARK-18114 related commit: {code:java} --conf spark.executor.extraJavaOptions="-Dfoo=\"first value\" -Dbar=another" {code} In current version submitting job with such extraJavaOptions causes following error: {code:java} Error: Unrecognized option: -Dbar=another Usage: spark-submit [options] [app arguments] Usage: spark-submit --kill [submission ID] --master [spark://...] Usage: spark-submit --status [submission ID] --master [spark://...] Usage: spark-submit run-example [options] example-class [example args] Options: --master MASTER_URL spark://host:port, mesos://host:port, yarn, or local. --deploy-mode DEPLOY_MODE Whether to launch the driver program locally ("client") or on one of the worker machines inside the cluster ("cluster") (Default: client). (... further spark-submit help ...) {code} Reverting SPARK-18114 is the solution to the issue. I can create a pull-request in GitHub. I thought about adding unit tests for that, buth methods generating driver launch command are private. was: Parameters passed to driver launching command in Mesos container are escaped using _shellEscape_ function. In SPARK-18114 additional wrapping in double quotes has been introduced. This cancels out quoting done by _shellEscape_ and makes in unable to run tasks with whitespaces in parameters, as they are interpreted as additional parameters to in-container spark-submit. This is how parameter passed to in-container spark-submit looks like now: {code:java} --conf "spark.executor.extraJavaOptions="-Dfoo=\"first value\" -Dbar=another"" {code} This is how they look after reverting SPARK-18114 related commit: {code:java} --conf spark.executor.extraJavaOptions="-Dfoo=\"first value\" -Dbar=another" {code} In current version submitting job with such extraJavaOptions causes following error: {code:java} Error: Unrecognized option: -Dbar=another Usage: spark-submit [options] [app arguments] Usage: spark-submit --kill [submission ID] --master [spark://...] Usage: spark-submit --status [submission ID] --master [spark://...] Usage: spark-submit run-example [options] example-class [example args] Options: --master MASTER_URL spark://host:port, mesos://host:port, yarn, or local. --deploy-mode DEPLOY_MODE Whether to launch the driver program locally ("client") or on one of the worker machines inside the cluster ("cluster") (Default: client). (... further spark-submit help ...) {code} > MesosClusterScheduler double-escapes parameters to bash command > --- > > Key: SPARK-23464 > URL: https://issues.apache.org/jira/browse/SPARK-23464 > Project: Spark > Issue Type: Bug > Components: Mesos >Affects Versions: 2.2.0 > Environment: Spark 2.2.0 with Mesosphere patches (but the problem > exists in main repo) > DC/OS 1.9.5 >Reporter: Marcin Kurczych >Priority: Major > > Parameters passed to driver launching command in Mesos container are escaped > using _shellEscape_ function. In SPARK-18114 additional wrapping in double > quotes has been introduced. This cancels out quoting done by _shellEscape_ > and makes in unable to run tasks with whitespaces in parameters, as they are > interpreted as additional parameters to in-container spark-submit. > This is how parameter passed to in-container spark-submit looks like now: > {code:java} > --conf "spark.executor.extraJavaOptions="-Dfoo=\"first value\" -Dbar=another"" > {code} > This is how they look after reverting SPARK-18114 related commit: > {code:java} > --conf spark.executor.extraJavaOptions="-Dfoo=\"first value\" -Dbar=another" > {code} > In current version submitting job with such extraJavaOptions causes following > error: > {code:java} > Error: Unrecognized option: -Dbar=another > Usage: spark-submit [options] [app arguments] > Usage: spark-submit --kill [submission ID] --master [spark://...] > Usage: spark-submit --status [submission ID] --master [spark://...] > Usage:
[jira] [Updated] (SPARK-23464) MesosClusterScheduler double-escapes parameters to bash command
[ https://issues.apache.org/jira/browse/SPARK-23464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcin Kurczych updated SPARK-23464: Description: Parameters passed to driver launching command in Mesos container are escaped using _shellEscape_ function. In SPARK-18114 additional wrapping in double quotes has been introduced. This cancels out quoting done by _shellEscape_ and makes in unable to run tasks with whitespaces in parameters, as they are interpreted as additional parameters to in-container spark-submit. This is how parameter passed to in-container spark-submit looks like now: {code:java} --conf "spark.executor.extraJavaOptions="-Dfoo=\"first value\" -Dbar=another"" {code} This is how they look after reverting SPARK-18114 related commit: {code:java} --conf spark.executor.extraJavaOptions="-Dfoo=\"first value\" -Dbar=another" {code} In current version submitting job with such extraJavaOptions causes following error: {code:java} Error: Unrecognized option: -Dbar=another Usage: spark-submit [options] [app arguments] Usage: spark-submit --kill [submission ID] --master [spark://...] Usage: spark-submit --status [submission ID] --master [spark://...] Usage: spark-submit run-example [options] example-class [example args] Options: --master MASTER_URL spark://host:port, mesos://host:port, yarn, or local. --deploy-mode DEPLOY_MODE Whether to launch the driver program locally ("client") or on one of the worker machines inside the cluster ("cluster") (Default: client). (... further spark-submit help ...) {code} was: Parameters passed to driver launching command in Mesos container are escaped using _shellEscape_ function. In SPARK-18114 additional wrapping in double quotes has been introduced. This cancels out quoting done by _shellEscape_ and makes in unable to run tasks with whitespaces in parameters, as they are interpreted as additional parameters to in-container spark-submit. This is how parameter passed to in-container spark-submit looks like now: {code} --conf "spark.executor.extraJavaOptions="-Dfoo=\"first value\" -Dbar=another"" {code} This is how they look after reverting SPARK-18114 related commit: {code} --conf spark.executor.extraJavaOptions="-Dfoo=\"first value\" -Dbar=another" {code} In current version submitting job with such extraJavaOptions causes following error: {code} Error: Unrecognized option: -Dfoo=another Usage: spark-submit [options] [app arguments] Usage: spark-submit --kill [submission ID] --master [spark://...] Usage: spark-submit --status [submission ID] --master [spark://...] Usage: spark-submit run-example [options] example-class [example args] Options: --master MASTER_URL spark://host:port, mesos://host:port, yarn, or local. --deploy-mode DEPLOY_MODE Whether to launch the driver program locally ("client") or on one of the worker machines inside the cluster ("cluster") (Default: client). (... further spark-submit help ...) {code} > MesosClusterScheduler double-escapes parameters to bash command > --- > > Key: SPARK-23464 > URL: https://issues.apache.org/jira/browse/SPARK-23464 > Project: Spark > Issue Type: Bug > Components: Mesos >Affects Versions: 2.2.0 > Environment: Spark 2.2.0 with Mesosphere patches (but the problem > exists in main repo) > DC/OS 1.9.5 >Reporter: Marcin Kurczych >Priority: Major > > Parameters passed to driver launching command in Mesos container are escaped > using _shellEscape_ function. In SPARK-18114 additional wrapping in double > quotes has been introduced. This cancels out quoting done by _shellEscape_ > and makes in unable to run tasks with whitespaces in parameters, as they are > interpreted as additional parameters to in-container spark-submit. > This is how parameter passed to in-container spark-submit looks like now: > {code:java} > --conf "spark.executor.extraJavaOptions="-Dfoo=\"first value\" -Dbar=another"" > {code} > This is how they look after reverting SPARK-18114 related commit: > {code:java} > --conf spark.executor.extraJavaOptions="-Dfoo=\"first value\" -Dbar=another" > {code} > In current version submitting job with such extraJavaOptions causes following > error: > {code:java} > Error: Unrecognized option: -Dbar=another > Usage: spark-submit [options] [app arguments] > Usage: spark-submit --kill [submission ID] --master [spark://...] > Usage: spark-submit --status [submission ID] --master [spark://...] > Usage: spark-submit run-example [options] example-class [example args] > Options: > --master MASTER_URL spark://host:port, mesos://host:port, yarn, or > local. > --deploy-mode DEPLOY_MODE Whether to launch the