[
https://issues.apache.org/jira/browse/SPARK-28525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcelo Vanzin resolved SPARK-28525.
------------------------------------
Resolution: Fixed
Fix Version/s: 3.0.0
Issue resolved by pull request 25265
[https://github.com/apache/spark/pull/25265]
> Allow Launcher to be applied Java options
> -----------------------------------------
>
> Key: SPARK-28525
> URL: https://issues.apache.org/jira/browse/SPARK-28525
> Project: Spark
> Issue Type: Improvement
> Components: Deploy
> Affects Versions: 3.0.0
> Reporter: Kousuke Saruta
> Assignee: Kousuke Saruta
> Priority: Minor
> Fix For: 3.0.0
>
>
> Launcher is implemented as a Java application and sometimes I'd like to apply
> Java options.
> One situation I have met is the time I try to attach debugger to Launcher.
> Launcher is launched from bin/spark-class but there is no room to apply Java
> options.
> {code}
> build_command() {
> "$RUNNER" -Xmx128m -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main
> "$@"
> printf "%d\0" $?
> }
> {code}
> Considering that it's not so many times to apply Java options to Launcher,
> one compromise would just modify spark-class by user like as follows.
> {code}
> build_command() {
> "$RUNNER" -Xmx128m $SPARK_LAUNCHER_OPTS -cp "$LAUNCH_CLASSPATH"
> org.apache.spark.launcher.Main "$@"
> printf "%d\0" $?
> }
> {code}
> But it doesn't work when any text related to Java options is output to
> standard output because whole output is used as command-string for
> spark-shell and spark-submit in current implementation.
> One example is jdwp. When apply agentlib option to use jdwp for debug, we
> will get output like as follows.
> {code}
> Listening for transport dt_socket at address: 9876
> {code}
> The output shown above is not a command-string so spark-submit and
> spark-shell will fail.
> To enable Java options for Launcher, we need treat command-string and others.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
