Re: Is it possible to obtain the full command to be invoked by SparkLauncher?
Thanks for the pointers. We figured out the stdout/stderr capture piece. I was just looking to capture the full command in order to help debug issues we run into with the submit depending on various combinations of all parameters/classpath, and also to isolate job specific issues from our wrapping application (i.e. being able to submit the job directly, rather than through our app). I will use the environment variable method for now. On Wed, Apr 24, 2019 at 4:18 PM Marcelo Vanzin wrote: > > BTW the SparkLauncher API has hooks to capture the stderr of the > spark-submit process into the logging system of the parent process. > Check the API javadocs since it's been forever since I looked at that. > > On Wed, Apr 24, 2019 at 1:58 PM Marcelo Vanzin wrote: > > > > Setting the SPARK_PRINT_LAUNCH_COMMAND env variable to 1 in the > > launcher env will make Spark code print the command to stderr. Not > > optimal but I think it's the only current option. > > > > On Wed, Apr 24, 2019 at 1:55 PM Jeff Evans > > wrote: > > > > > > The org.apache.spark.launcher.SparkLauncher is used to construct a > > > spark-submit invocation programmatically, via a builder pattern. In > > > our application, which uses a SparkLauncher internally, I would like > > > to log the full spark-submit command that it will invoke to our log > > > file, in order to aid in debugging/support. However, I can't figure > > > out a way to do this. This snippet would work, except for the fact > > > that the createBuilder method is private. > > > > > > sparkLauncher.createBuilder().command() > > > > > > Is there an alternate way of doing this? The Spark version is > > > 2.11:2.4.0. Thanks. > > > > > > - > > > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > > > > > > > > > -- > > Marcelo > > > > -- > Marcelo - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
Re: Is it possible to obtain the full command to be invoked by SparkLauncher?
BTW the SparkLauncher API has hooks to capture the stderr of the spark-submit process into the logging system of the parent process. Check the API javadocs since it's been forever since I looked at that. On Wed, Apr 24, 2019 at 1:58 PM Marcelo Vanzin wrote: > > Setting the SPARK_PRINT_LAUNCH_COMMAND env variable to 1 in the > launcher env will make Spark code print the command to stderr. Not > optimal but I think it's the only current option. > > On Wed, Apr 24, 2019 at 1:55 PM Jeff Evans > wrote: > > > > The org.apache.spark.launcher.SparkLauncher is used to construct a > > spark-submit invocation programmatically, via a builder pattern. In > > our application, which uses a SparkLauncher internally, I would like > > to log the full spark-submit command that it will invoke to our log > > file, in order to aid in debugging/support. However, I can't figure > > out a way to do this. This snippet would work, except for the fact > > that the createBuilder method is private. > > > > sparkLauncher.createBuilder().command() > > > > Is there an alternate way of doing this? The Spark version is > > 2.11:2.4.0. Thanks. > > > > - > > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > > > > > -- > Marcelo -- Marcelo - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
Re: Is it possible to obtain the full command to be invoked by SparkLauncher?
You could set the env var SPARK_PRINT_LAUNCH_COMMAND and spark-submit will print it, but it will be printed by the subprocess and not yours unless you redirect the stdout Also the command is what spark-submit generates, so it is quite more verbose and includes the classpath etc. I think the only alternative if the above is not enough is to get hold of the builder - you might need to extend the launcher and push it on the same package since On Wed, 24 Apr 2019 at 21:55, Jeff Evans wrote: > The org.apache.spark.launcher.SparkLauncher is used to construct a > spark-submit invocation programmatically, via a builder pattern. In > our application, which uses a SparkLauncher internally, I would like > to log the full spark-submit command that it will invoke to our log > file, in order to aid in debugging/support. However, I can't figure > out a way to do this. This snippet would work, except for the fact > that the createBuilder method is private. > > sparkLauncher.createBuilder().command() > > Is there an alternate way of doing this? The Spark version is > 2.11:2.4.0. Thanks. > > - > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
Re: Is it possible to obtain the full command to be invoked by SparkLauncher?
Setting the SPARK_PRINT_LAUNCH_COMMAND env variable to 1 in the launcher env will make Spark code print the command to stderr. Not optimal but I think it's the only current option. On Wed, Apr 24, 2019 at 1:55 PM Jeff Evans wrote: > > The org.apache.spark.launcher.SparkLauncher is used to construct a > spark-submit invocation programmatically, via a builder pattern. In > our application, which uses a SparkLauncher internally, I would like > to log the full spark-submit command that it will invoke to our log > file, in order to aid in debugging/support. However, I can't figure > out a way to do this. This snippet would work, except for the fact > that the createBuilder method is private. > > sparkLauncher.createBuilder().command() > > Is there an alternate way of doing this? The Spark version is > 2.11:2.4.0. Thanks. > > - > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > -- Marcelo - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
