Thanks for the pointers. We figured out the stdout/stderr capture piece. I was just looking to capture the full command in order to help debug issues we run into with the submit depending on various combinations of all parameters/classpath, and also to isolate job specific issues from our wrapping application (i.e. being able to submit the job directly, rather than through our app). I will use the environment variable method for now.
On Wed, Apr 24, 2019 at 4:18 PM Marcelo Vanzin <van...@cloudera.com> wrote: > > BTW the SparkLauncher API has hooks to capture the stderr of the > spark-submit process into the logging system of the parent process. > Check the API javadocs since it's been forever since I looked at that. > > On Wed, Apr 24, 2019 at 1:58 PM Marcelo Vanzin <van...@cloudera.com> wrote: > > > > Setting the SPARK_PRINT_LAUNCH_COMMAND env variable to 1 in the > > launcher env will make Spark code print the command to stderr. Not > > optimal but I think it's the only current option. > > > > On Wed, Apr 24, 2019 at 1:55 PM Jeff Evans > > <jeffrey.wayne.ev...@gmail.com> wrote: > > > > > > The org.apache.spark.launcher.SparkLauncher is used to construct a > > > spark-submit invocation programmatically, via a builder pattern. In > > > our application, which uses a SparkLauncher internally, I would like > > > to log the full spark-submit command that it will invoke to our log > > > file, in order to aid in debugging/support. However, I can't figure > > > out a way to do this. This snippet would work, except for the fact > > > that the createBuilder method is private. > > > > > > sparkLauncher.createBuilder().command() > > > > > > Is there an alternate way of doing this? The Spark version is > > > 2.11:2.4.0. Thanks. > > > > > > --------------------------------------------------------------------- > > > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > > > > > > > > > -- > > Marcelo > > > > -- > Marcelo --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
