jerryshao opened a new pull request, #14230: URL: https://github.com/apache/gravitino/pull/14230
### What changes were proposed in this pull request? Follow-ups of the `k8s` job executor (#13613, #13616, #13617). **Harden the k8s job executor** - A driver pod whose phase is `Succeeded` or `Failed` has terminated, whatever its container status says, for example after its node was lost. Only otherwise the driver container decides, as before. - A create that fails with a 5xx gets the same best-effort cleanup as one without a response: the SparkApplication may have been created. If the cleanup fails, a WARN tells how to delete it by hand. Gravitino doesn't sweep the SparkApplications of jobs it doesn't know (see #13554). - The SparkApplications of a namespace are listed page by page (500 per request). A listing that times out waiting for the response no longer counts as an unreachable API server: the jobs are got one by one instead of failing every status query for 10 s. - Where a read decides the deletion (the no-status cleanup and the failed-submit cleanup), the SparkApplication is deleted with a `resourceVersion` precondition and the `Foreground` propagation, and read again on a 409 (up to 3 retries). An application that got a status meanwhile follows the normal status path. fabric8 7.2.0 types `lockResourceVersion()` as `ReplaceDeletable`, without `withPropagationPolicy`, so the resource is used as the `ExtensibleResource` it is; both end up in the `DeleteOptions` of the request. **Job status poll** - `JobManager` logged an ERROR with a full stack trace for every active job on every poll while the job executor failed, for example while the API server was unreachable. It now logs one WARN per poll and metalake, with the number of failed jobs, the first five of them and the first failure, and each failed job at DEBUG. **Output size** - `JobExecutor#getJobStdout`/`getJobStderr`, `gravitino.job.outputMaxBytes` and the docs said `maxBytes` bounds both the read cost and the response size. It bounds the response size; an executor may not bound the bytes it reads, as the k8s job executor reads the last lines of the driver log. ### Why are the changes needed? They are the follow-ups of the k8s job executor collected in #13554: a terminated driver pod could be deleted or kept by mistake, a SparkApplication could run unknown after a create that failed with a 5xx, a namespace with many SparkApplications could stop all status updates, a deletion could act on a SparkApplication that changed after it was read, and the status poll flooded the log while the API server was unreachable. Fix: #14058 ### Does this PR introduce _any_ user-facing change? - The status poll failures are logged once per poll and metalake at WARN instead of once per job at ERROR. - The description of `gravitino.job.outputMaxBytes` changed. No new configuration. ### How was this patch tested? - Unit tests with the fabric8 mock server for the submit cleanup after a 5xx (also of a resource of another job, and after a conflict), the logs of a deleted SparkApplication, the paged listing, a slow listing, the conditional deletion of a job without status (request body, a status written meanwhile, retries running out), `isDriverRunning`, the cache eviction, and the summarized poll log in `TestJobManager`. The key tests fail when their fix is reverted. - Manually against a local Kubernetes cluster (OrbStack) with spark-kubernetes-operator 0.9.0: Spark Pi succeeded with its driver output, a running job was cancelled, a SparkApplication that never got a status (in a namespace the operator doesn't watch) was deleted and its job failed, and the API server rejected a `Foreground` deletion with a stale `resourceVersion` precondition with a 409 and accepted it with the current one. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
