[
https://issues.apache.org/jira/browse/OOZIE-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16101473#comment-16101473
]
Peter Cseh commented on OOZIE-2984:
-----------------------------------
Thanks for the update!
It looks fine in general, but it does not work correctly if --verbose is in the
list. This option has no value for it an he current code adds an empty string
to the parameter list. I don't think Spark likes that.
Also, I don't know if this is the only option that has no parameters or are
there more out there, so this should be solved in a general way.
After this, DG_SparkActionExtension.twiki should be updated to say the quoting
is not a hard requirement any more. Right now it says {quote}Values containing
whitespaces should be enclosed by double quotes. {quote}
> Parse spark-defaults.conf values with spaces without needing the quotes
> -----------------------------------------------------------------------
>
> Key: OOZIE-2984
> URL: https://issues.apache.org/jira/browse/OOZIE-2984
> Project: Oozie
> Issue Type: Bug
> Affects Versions: 4.3.0
> Reporter: Fahd Siddiqui
> Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: OOZIE-2984.001.patch, OOZIE-2984.002.patch,
> OOZIE-2984.003.patch
>
>
> Oozie requires multiple java opts to be enclosed in quotes. For example:
> {code}
> spark.driver.extraJavaOptions="-Xmn2703m -XX:SurvivorRatio=2
> -XX:ParallelGCThreads=20"
> {code}
> However the above breaks spark-shell.
> {code}
> $ ./spark2-shell
> Invalid initial young generation size: -Xmn2212m -XX:SurvivorRatio=2
> -XX:ParallelGCThreads=20
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
> {code}
> To fix spark-shell, we have to remove the quotes, which errors out Oozie:
> {code}
> Error: Unrecognized option '-XX:SurvivorRatio=2'
> ...
> --conf
> spark.executor.extraJavaOptions=-Xmn2703m
> -Dlog4j.configuration=spark-log4j.properties
> -XX:SurvivorRatio=2
> {code}
> Oozie should be able to parse spark-defaults.conf values with spaces without
> needing the quotes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
