jerryshao opened a new pull request, #14047:
URL: https://github.com/apache/gravitino/pull/14047
### What changes were proposed in this pull request?
When a node-local job owned by another executor is still `CANCELLING` three
status pull intervals after the cancel request, the server pulling job statuses
marks it as `CANCELLED`.
- **`JobManager`**:
- In `pullAndUpdateJobStatus`, a job this server doesn't own is no longer
always skipped. If the executor is node-local, the job is `CANCELLING`, and its
last update is older than 3 × `gravitino.job.statusPullIntervalInMs`, it is
marked as `CANCELLED` and a warning is logged.
- Like `expireStaleActiveJob`, the condition is checked again on the
entity that the update re-reads. A job the owner settled in the meantime is
left as it is.
- **Docs**:
- Cancelling a job left behind by an exited or restarted server now clears
it, and its job template can then be deleted.
- A new caution states the assumptions: every server uses the same pull
interval, and the servers' clocks are in sync.
### Why are the changes needed?
The local job executor gets a new identity on every server start, so a
restarted server no longer owns the jobs it ran before. Cancelling such a job
only marks it as `CANCELLING` for an owner that is gone. The job then stays
`CANCELLING`, and its job template can't be deleted, until it expires after
`gravitino.job.stagingDirKeepTimeInMs` (about 7.7 days by default).
A live owner answers a cancellation within about two pull intervals: it
cancels the job at its next pull, and records it as `CANCELLED` at the pull
after the process exits. So a job still `CANCELLING` after three intervals has
most likely been left behind, and cancelling across servers keeps working. When
several servers settle the same job at once, OCC lets only one write through.
The others are deferred to the next pull, which then sees the job as finished.
Fix: #14046
### Does this PR introduce _any_ user-facing change?
Yes. With the local job executor, a job that another server owns and that
has been `CANCELLING` for more than 3 × `gravitino.job.statusPullIntervalInMs`
is now marked as `CANCELLED`. If its owner is still alive but doesn't answer in
time, the job's process may keep running. That can happen when the servers use
different pull intervals or their clocks are off, when the owner can't pull for
that long, or when the process takes longer than one interval to exit. The docs
describe these cases.
There are no API or configuration changes.
### How was this patch tested?
- **`TestJobManager`**:
- Settles a stale unanswered cancellation, including a job whose last
modified time is unset.
- Leaves a recent cancellation alone, and a job owned by another server
that isn't `CANCELLING`.
- Leaves jobs of non-node-local executors alone.
- Keeps a job that its owner settled concurrently.
- Tolerates an OCC conflict, and the next pull leaves the job as it is.
- **`TestJobManagerMultiNode`**:
- Node A exits and node B cancels its job. The job stays `CANCELLING` and
blocks deleting the template until the timeout, then node B marks it as
`CANCELLED` and the template can be deleted.
- A pull on node B before the timeout doesn't affect a cancellation that
the live node A then carries out.
- **`JobIT`** (embedded): a job of another server that is cancelled becomes
`CANCELLED`, and its template can then be deleted.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
