Re: Is it possible to obtain the full command to be invoked by SparkLauncher?
Thanks for the pointers. We figured out the stdout/stderr capture piece. I was just looking to capture the full command in order to help debug issues we run into with the submit depending on various combinations of all parameters/classpath, and also to isolate job specific issues from our wrapping application (i.e. being able to submit the job directly, rather than through our app). I will use the environment variable method for now. On Wed, Apr 24, 2019 at 4:18 PM Marcelo Vanzin wrote: > > BTW the SparkLauncher API has hooks to capture the stderr of the > spark-submit process into the logging system of the parent process. > Check the API javadocs since it's been forever since I looked at that. > > On Wed, Apr 24, 2019 at 1:58 PM Marcelo Vanzin wrote: > > > > Setting the SPARK_PRINT_LAUNCH_COMMAND env variable to 1 in the > > launcher env will make Spark code print the command to stderr. Not > > optimal but I think it's the only current option. > > > > On Wed, Apr 24, 2019 at 1:55 PM Jeff Evans > > wrote: > > > > > > The org.apache.spark.launcher.SparkLauncher is used to construct a > > > spark-submit invocation programmatically, via a builder pattern. In > > > our application, which uses a SparkLauncher internally, I would like > > > to log the full spark-submit command that it will invoke to our log > > > file, in order to aid in debugging/support. However, I can't figure > > > out a way to do this. This snippet would work, except for the fact > > > that the createBuilder method is private. > > > > > > sparkLauncher.createBuilder().command() > > > > > > Is there an alternate way of doing this? The Spark version is > > > 2.11:2.4.0. Thanks. > > > > > > - > > > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > > > > > > > > > -- > > Marcelo > > > > -- > Marcelo - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
Re: Is it possible to obtain the full command to be invoked by SparkLauncher?
BTW the SparkLauncher API has hooks to capture the stderr of the spark-submit process into the logging system of the parent process. Check the API javadocs since it's been forever since I looked at that. On Wed, Apr 24, 2019 at 1:58 PM Marcelo Vanzin wrote: > > Setting the SPARK_PRINT_LAUNCH_COMMAND env variable to 1 in the > launcher env will make Spark code print the command to stderr. Not > optimal but I think it's the only current option. > > On Wed, Apr 24, 2019 at 1:55 PM Jeff Evans > wrote: > > > > The org.apache.spark.launcher.SparkLauncher is used to construct a > > spark-submit invocation programmatically, via a builder pattern. In > > our application, which uses a SparkLauncher internally, I would like > > to log the full spark-submit command that it will invoke to our log > > file, in order to aid in debugging/support. However, I can't figure > > out a way to do this. This snippet would work, except for the fact > > that the createBuilder method is private. > > > > sparkLauncher.createBuilder().command() > > > > Is there an alternate way of doing this? The Spark version is > > 2.11:2.4.0. Thanks. > > > > - > > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > > > > > -- > Marcelo -- Marcelo - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
Re: Is it possible to obtain the full command to be invoked by SparkLauncher?
You could set the env var SPARK_PRINT_LAUNCH_COMMAND and spark-submit will print it, but it will be printed by the subprocess and not yours unless you redirect the stdout Also the command is what spark-submit generates, so it is quite more verbose and includes the classpath etc. I think the only alternative if the above is not enough is to get hold of the builder - you might need to extend the launcher and push it on the same package since On Wed, 24 Apr 2019 at 21:55, Jeff Evans wrote: > The org.apache.spark.launcher.SparkLauncher is used to construct a > spark-submit invocation programmatically, via a builder pattern. In > our application, which uses a SparkLauncher internally, I would like > to log the full spark-submit command that it will invoke to our log > file, in order to aid in debugging/support. However, I can't figure > out a way to do this. This snippet would work, except for the fact > that the createBuilder method is private. > > sparkLauncher.createBuilder().command() > > Is there an alternate way of doing this? The Spark version is > 2.11:2.4.0. Thanks. > > - > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
Re: Is it possible to obtain the full command to be invoked by SparkLauncher?
Setting the SPARK_PRINT_LAUNCH_COMMAND env variable to 1 in the launcher env will make Spark code print the command to stderr. Not optimal but I think it's the only current option. On Wed, Apr 24, 2019 at 1:55 PM Jeff Evans wrote: > > The org.apache.spark.launcher.SparkLauncher is used to construct a > spark-submit invocation programmatically, via a builder pattern. In > our application, which uses a SparkLauncher internally, I would like > to log the full spark-submit command that it will invoke to our log > file, in order to aid in debugging/support. However, I can't figure > out a way to do this. This snippet would work, except for the fact > that the createBuilder method is private. > > sparkLauncher.createBuilder().command() > > Is there an alternate way of doing this? The Spark version is > 2.11:2.4.0. Thanks. > > - > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > -- Marcelo - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
Is it possible to obtain the full command to be invoked by SparkLauncher?
The org.apache.spark.launcher.SparkLauncher is used to construct a spark-submit invocation programmatically, via a builder pattern. In our application, which uses a SparkLauncher internally, I would like to log the full spark-submit command that it will invoke to our log file, in order to aid in debugging/support. However, I can't figure out a way to do this. This snippet would work, except for the fact that the createBuilder method is private. sparkLauncher.createBuilder().command() Is there an alternate way of doing this? The Spark version is 2.11:2.4.0. Thanks. - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
