jerryshao opened a new issue, #13617:
URL: https://github.com/apache/gravitino/issues/13617
### Describe the subtask
The `k8s` job executor added in #13613 rejects job templates with
environment variables, because writing them into the SparkApplication would
expose values such as credentials in plain text. It also submits every job
right away, so a burst of submissions can flood the cluster, and many jobs then
fail on the operator's start timeouts.
- Environment variables:
- Put the environment variables of a job into a Secret whose owner is the
SparkApplication, so that Kubernetes deletes the Secret with it.
- Pass the Secret to the driver and executors through
`spark.kubernetes.driver.secretKeyRef.*` and
`spark.kubernetes.executor.secretKeyRef.*`.
- If the Secret can't be created, delete the SparkApplication and fail the
submission.
- Active job limit:
- Add `maxActiveJobs`, the maximum number of unfinished jobs in a
namespace. It is unlimited by default.
- Before submitting a job, count the unfinished jobs in its namespace with
the status cache. If the limit is reached, reject the job with a clear error so
that the caller can retry later.
- Counting per namespace needs no permission beyond the namespaces the
jobs run in. Across Gravitino servers the count is approximate.
### Parent issue
#13554
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
