jerryshao opened a new pull request, #13132:
URL: https://github.com/apache/gravitino/pull/13132
### What changes were proposed in this pull request?
Reject a Spark job at submission when the local job executor cannot launch
it, instead of accepting it and failing it asynchronously.
- `SparkProcessBuilder`: extract the `sparkHome`/`SPARK_HOME` and
`spark-submit` checks into `resolveSparkSubmit(configs)`, and replace the
mistaken `org.apache.arrow.util` imports with Guava.
- `LocalJobExecutor`:
- `submitJob()` validates Spark templates before queueing and throws
`IllegalArgumentException` if Spark is not available.
- `initialize()` logs a warning when Spark is not available. Server
startup is not blocked, since Spark is optional for the local executor.
- `JobManager.runJob()`:
- Rethrows `IllegalArgumentException` from the executor as is, so the REST
API returns 400 with the original reason instead of a generic 500.
- Removes the staging directory of a job whose submission fails.
- Docs: update `manage-jobs-in-gravitino.md` and document the 400 response
of `runJob` in `docs/open-api/jobs.yaml`.
### Why are the changes needed?
With `sparkHome` and `SPARK_HOME` unset, a Spark job run request succeeds
and the job stays `QUEUED` until the next status poll, then flips to `FAILED`.
The reason appears only in the server log, so an operator using the API or UI
cannot tell which setting is missing.
Fix: #13131
### Does this PR introduce _any_ user-facing change?
Yes. Running a Spark job with the local job executor when Spark is not
available now returns 400 with the reason, and no job is created. Previously
the request succeeded and the job failed later without a visible reason. No API
or configuration keys are added or removed.
### How was this patch tested?
- Unit tests:
- `TestSparkProcessBuilder#testResolveSparkSubmit`
-
`TestLocalJobExecutor#testSubmitSparkJobRejectedWhenSparkSubmitIsNotAvailable`
- `TestJobManager#testRunJobPropagatesJobExecutorRejection`
- Integration test `JobIT#testRunSparkJobRejectedWhenSparkIsNotAvailable`
covers the client → REST → executor path. It verifies the rejection reason
reaches the client, no job or staging directory is left behind, and shell jobs
are unaffected.
- `./gradlew :docs:build` passes for the OpenAPI change.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
