GitHub user vanzin opened a pull request:
https://github.com/apache/spark/pull/20925
[SPARK-22941][core] Do not exit JVM when submit fails with in-process
launcher.
The current in-process launcher implementation just calls the SparkSubmit
object, which, in case of errors, will more often than not exit the JVM.
This is not desirable since this launcher is meant to be used inside other
applications, and that would kill the application.
The change turns SparkSubmit into a class, and abstracts aways some of
the functionality used to print error messages and abort the submission
process. The default implementation uses the logging system for messages,
and throws exceptions for errors. As part of that I also moved some code
that doesn't really belong in SparkSubmit to a better location.
The command line invocation of spark-submit now uses a special
implementation
of the SparkSubmit class that overrides those behaviors to do what is
expected
from the command line version (print to the terminal, exit the JVM, etc).
A lot of the changes are to replace calls to methods such as
"printErrorAndExit"
with the new API.
As part of adding tests for this, I had to fix some small things in the
launcher option parser so that things like "--version" can work when
used in the launcher library.
There is still code that prints directly to the terminal, like all the
Ivy-related code in SparkSubmitUtils, and other areas where some
re-factoring
would help, like the CommandLineUtils class, but I chose to leave those
alone to keep this change more focused.
Aside from existing and added unit tests, I ran command line tools with
a bunch of different arguments to make sure messages and errors behave
like before.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vanzin/spark SPARK-22941
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20925.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20925
----
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
