LuciferYang opened a new pull request, #13679: URL: https://github.com/apache/gravitino/pull/13679
### What changes were proposed in this pull request? `LocalJobExecutor.cancelJob` still sends SIGTERM first, and now also schedules a check that kills the process forcibly if it is still alive after `gravitino.jobExecutor.local.cancelForceKillDelayInMs` (30 seconds by default). `runJob` now checks whether the job was being cancelled before it looks at the exit code, so a cancelled job ends CANCELLED whatever code its process exits with. The new setting is added to the local executor configuration table in `docs/manage-jobs-in-gravitino.md`. Like the existing SIGTERM, the forced kill reaches only the job's own process, not processes it started. ### Why are the changes needed? A cancelled job whose process ignored SIGTERM never exited: it stayed CANCELLING and kept its worker thread blocked in `waitFor()`. The worker pool is fixed at `maxRunningJobs`, so each such job took a slot for good, and once all slots were held new jobs stayed queued until a restart. Separately, a cancelled job whose process trapped SIGTERM and exited 0 was reported SUCCEEDED, because the exit code was checked first. Fix: #13678 ### Does this PR introduce _any_ user-facing change? Yes. A new setting, `gravitino.jobExecutor.local.cancelForceKillDelayInMs` (default `30000`), controls how long a cancelled job's process may keep running before it is killed forcibly. A cancelled job that exits 0 is now reported CANCELLED instead of SUCCEEDED. ### How was this patch tested? Added `testCancelJobEscalatesToForcibleKill`, which cancels a job that ignores SIGTERM and asserts it reaches CANCELLED once the force-kill delay passes, and `testCancelledJobExitingZeroReportsCancelled`, which cancels a job that traps SIGTERM and exits 0 and asserts CANCELLED. Each script prints a marker after installing its trap, and the test waits for it before cancelling so the signal cannot arrive first. Both tests fail against the code before this change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
