CryoThrust commented on issue #12119: URL: https://github.com/apache/seatunnel/issues/12119#issuecomment-5556420197
For the two-phase boundary, I suggest treating the acceptance record as the source of truth and making the heavy `JobMaster` strictly reconstructible from it: - Persist an immutable record before `JobMaster.init()` with `jobId`, config/artifact digests, derived resource descriptor (pipeline/slot requirements), enqueue sequence, priority, engine version and `masterEpoch`. The record should have an explicit lifecycle such as `ACCEPTED -> INITIALIZING -> RUNNING` (or `FAILED/CANCELLED`). - A cancellation of an `ACCEPTED` job should atomically mark a tombstone/version before initialization starts. Restore and the admission worker must check that version so a late initializer cannot resurrect a cancelled job. - If a queued job is re-planned when admitted, compare the resulting resource descriptor and digests with the acceptance record. A mismatch should transition to a visible initialization failure, not silently change scheduling order or resource accounting. - Keep configuration/plugin-load errors in the job lifecycle for deferred initialization, while preserving the submit acknowledgement as “accepted for scheduling”; callers should not interpret it as “configuration initialized successfully”. - On master restore, rebuild only the lightweight record/index first, then let the admission loop construct plans in persisted queue order. This provides a clean hand-off to #12120 rather than reintroducing eager `JobMaster` creation during recovery. The contract tests should cover cancel-before-init, duplicate admission after failover, digest/version mismatch on re-plan, and an initializer finishing after a newer cancellation/update. These cases are where the two-phase design can otherwise leak a queued job back into the running set. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
