Kousuke Saruta created SPARK-28525:
--------------------------------------
Summary: Allow Launcher to be applied Java options
Key: SPARK-28525
URL: https://issues.apache.org/jira/browse/SPARK-28525
Project: Spark
Issue Type: Improvement
Components: Deploy
Affects Versions: 3.0.0
Reporter: Kousuke Saruta
Assignee: Kousuke Saruta
Launcher is implemented as a Java application and sometimes I'd like to apply
Java options.
One situation I have met is the time I try to attach debugger to Launcher.
Launcher is launched from bin/spark-class but there is no room to apply Java
options.
{code}
build_command() {
"$RUNNER" -Xmx128m -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main "$@"
printf "%d\0" $?
}
{code}
Considering that it's not so many times to apply Java options to Launcher, one
compromise would just modify spark-class by user like as follows.
{code}
build_command() {
"$RUNNER" -Xmx128m $SPARK_LAUNCHER_OPTS -cp "$LAUNCH_CLASSPATH"
org.apache.spark.launcher.Main "$@"
printf "%d\0" $?
}
{code}
But it doesn't work when any text related to Java options is output to standard
output because whole output is used as command-string for spark-shell and
spark-submit in current implementation.
One example is jdwp. When apply agentlib option to use jdwp for debug, we will
get output like as follows.
{code}
Listening for transport dt_socket at address: 9876
{code}
The output shown above is not a command-string so spark-submit and spark-shell
will fail.
To enable Java options for Launcher, we need treat command-string and others.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
