[
https://issues.apache.org/jira/browse/OOZIE-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083680#comment-16083680
]
Andras Piros edited comment on OOZIE-2984 at 7/12/17 9:09 AM:
--------------------------------------------------------------
[~gezapeti] such quoting scenario is possible when calling {{spark-submit}}:
{noformat}
spark-submit \
...
--conf \
spark.executor.extraJavaOptions="-Dlog4j.configuration=spark-log4j.properties
-Dsome.executor.key=\"some value\"" \
--conf \
spark.driver.extraJavaOptions="-Dsome.driver.key=some.value
-Dlog4j.configuration=spark-log4j.properties" \
...
{noformat}
Producing following output:
{noformat}
Using properties file: /usr/lib/spark/conf/spark-defaults.conf
...
driverExtraJavaOptions -Dsome.driver.key=some.value
-Dlog4j.configuration=spark-log4j.properties
...
Spark properties used, including those specified through
--conf and those from the properties file
/usr/lib/spark/conf/spark-defaults.conf:
...
(spark.executor.extraJavaOptions,-Dlog4j.configuration=spark-log4j.properties
-Dsome.executor.key="some value")
...
(spark.driver.extraJavaOptions,-Dsome.driver.key=some.value
-Dlog4j.configuration=spark-log4j.properties)
...
System properties:
...
(spark.executor.extraJavaOptions,-Dlog4j.configuration=spark-log4j.properties
-Dsome.executor.key="some value")
...
(spark.driver.extraJavaOptions,-Dsome.driver.key=some.value
-Dlog4j.configuration=spark-log4j.properties)
...
{noformat}
Here we can see it's possible to (both are understood by {{spark-submit}}, so
possibly used by Spark users):
* quote a {{--conf}} option fully to have multiple options within an option
* escape-quote a sub-option inside an already quoted {{--conf}} option
was (Author: andras.piros):
[~gezapeti] such quoting scenario is possible when calling {{spark-submit}}:
{noformat}
spark-submit \
...
--conf \
spark.executor.extraJavaOptions="-Dlog4j.configuration=spark-log4j.properties
-Dsome.executor.key=\"some value\"" \
--conf \
spark.driver.extraJavaOptions="-Dsome.driver.key=some.value
-Dlog4j.configuration=spark-log4j.properties" \
...
{noformat}
Producing following output:
{noformat}
Using properties file: /usr/lib/spark/conf/spark-defaults.conf
...
driverExtraJavaOptions -Dsome.driver.key=some.value
-Dlog4j.configuration=spark-log4j.properties
...
Spark properties used, including those specified through
--conf and those from the properties file
/usr/lib/spark/conf/spark-defaults.conf:
...
(spark.executor.extraJavaOptions,-Dlog4j.configuration=spark-log4j.properties
-Dsome.executor.key="some value")
...
(spark.driver.extraJavaOptions,-Dsome.driver.key=some.value
-Dlog4j.configuration=spark-log4j.properties)
...
System properties:
...
(spark.executor.extraJavaOptions,-Dlog4j.configuration=spark-log4j.properties
-Dsome.executor.key="some value")
...
(spark.driver.extraJavaOptions,-Dsome.driver.key=some.value
-Dlog4j.configuration=spark-log4j.properties)
...
{noformat}
Here we can see it's possible to (both are understood by {{spark-shell}}, so
possibly used by Spark users):
* quote a {{--conf}} option fully to have multiple options within an option
* escape-quote a sub-option inside an already quoted {{--conf}} option
> Parse spark-defaults.conf values with spaces without needing the quotes.
> ------------------------------------------------------------------------
>
> Key: OOZIE-2984
> URL: https://issues.apache.org/jira/browse/OOZIE-2984
> Project: Oozie
> Issue Type: Bug
> Affects Versions: trunk
> Reporter: Fahd Siddiqui
> Assignee: Andras Piros
>
> Oozie requires multiple java opts to be enclosed in quotes. For example:
> {code}
> spark.driver.extraJavaOptions="-Xmn2703m -XX:SurvivorRatio=2
> -XX:ParallelGCThreads=20"
> {code}
> However the above breaks spark-shell.
> {code}
> $ ./spark2-shell
> Invalid initial young generation size: -Xmn2212m -XX:SurvivorRatio=2
> -XX:ParallelGCThreads=20
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
> {code}
> To fix spark-shell, we have to remove the quotes, which errors out Oozie:
> {code}
> Error: Unrecognized option '-XX:SurvivorRatio=2'
> ...
> --conf
> spark.executor.extraJavaOptions=-Xmn2703m
> -Dlog4j.configuration=spark-log4j.properties
> -XX:SurvivorRatio=2
> {code}
> Oozie should be able to parse spark-defaults.conf values with spaces without
> needing the quotes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
