sarutak opened a new pull request #25265: [SPARK-28525][DEPLOY] Allow Launcher to be applied Java options URL: https://github.com/apache/spark/pull/25265 Launcher is implemented as a Java application and sometimes I'd like to apply Java options. One situation I have met is the time I try to attach debugger to Launcher. Launcher is launched from bin/spark-class but there is no room to apply Java options. ``` build_command() { "$RUNNER" -Xmx128m -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main "$@" printf "%d\0" $? } ``` Considering that it's not so many times to apply Java options to Launcher, one compromise would just modify spark-class by user like as follows. ``` build_command() { "$RUNNER" -Xmx128m $SPARK_LAUNCHER_OPTS -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main "$@" printf "%d\0" $? } ``` But it doesn't work when any text related to Java options is output to standard output because whole output is used as command-string for spark-shell and spark-submit in current implementation. One example is jdwp. When apply agentlib option to use jdwp for debug, we will get output like as follows. ``` Listening for transport dt_socket at address: 9876 ``` The output shown above is not a command-string so spark-submit and spark-shell will fail. To enable Java options for Launcher, we need treat command-string and others. I changed launcher/Main.java and bin/spark-class to print separator-character and treat it. ## How was this patch tested? Tested manually using Spark Shell with / without LAUNCHER_JAVA_OPTIONS like as follows. ``` SPARK_LAUNCHER_OPTS="-agentlib:jdwp=transport=dt_socket,suspend=y,address=localhost:9876,server=y" bin/spark-shell ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
