goutamadwant commented on issue #11735:
URL: https://github.com/apache/seatunnel/issues/11735#issuecomment-5826654245
@DanielLeens @SEZ9 Thanks. I updated #11734 (`a986eb16`) and the STIP body
above. The rules you asked for are now tables in a new "Normative Rules"
section, and the top of the doc lists what changed since `e944f1e3`.
1. **Attempt key.**
- R1 gives the value copied at each moment: constructor, adoption,
restore (including a retried write and a null slot) and deployment.
`INITIALIZING` is not the key.
- It also names the single writer: the active master's `SubPlan`, whose
`resetPipelineState()` runs inside `synchronized reset()` under `restoreLock`.
- R2 resolves every kind of report, including duplicates and unknown
`executionId`s. No rule compares the clocks of different masters.
- Criteria 11–13 and 34.
2. **Dropped operations.**
- R3 lists each operation that can be dropped or rejected by the fence
(`NO_ENTRY`, `OWNER_MISMATCH`, `TERMINAL`). For each, it says what history,
REST and the `/job-info` diagnostics then show.
- A dropped record is simply absent and logged at WARN. The diagnostics
never read failure history, so they are unchanged. History does not claim to be
complete.
- Ordering statements now cover only operations the queue accepted, and
the consumer retries in place before taking the next operation. The terminal
fence depends only on the owner, the terminal flag and `terminalTime`.
- Restore and cleanup never wait on the queue.
- Criteria 20 and 35.
3. **Wire compatibility.**
- #12466 pins both UIDs and adds `TaskStateSerializationTest`. Its
fixtures were written by the unmodified classes on current `dev`, with
identical bytes on JDK 8 and 11.
- The test asserts that the fixtures deserialize, that the pinned classes
write the same bytes, and that the UIDs match. It adds no field.
- The slice that adds fields keeps these fixtures and adds the old/new
direction tests (Compatibility).
4. **Maps.**
- R4 lists each history map and the in-memory state with its writers,
fence, expiry, cleanup owner and IMap storage.
- A finished snapshot only replaces one whose owner has a smaller
`initializationTimestamp`. Only an orphan, the entry of a submission whose job
never reached an end state, is removed without writing a snapshot.
- Criteria 22, 24, 36 and 37.
From the review on #11734:
- **REST order.** It is now decision table R5. It checks for an absent state
before calling `isEndState()`, and never treats an absent state as running.
- **Create before capture.** `JobMaster.init` offers create (or adopt after
failover) right after the plan is built, before `initCheckPointManager` and
`initStateFuture`, which can capture master-recovery failures. `init` runs
before the job enters the pending queue, both on submit and after failover. A
later operation that finds no entry is rejected and logged as `NO_ENTRY`
(Ownership, R3 and R4).
Terminal handling states one limit. If a savepoint start's create operation
is dropped, or its snapshot write fails until its entry expires, then after
that run finishes REST can still return the previous incarnation's snapshot
until that snapshot's TTL expires. Otherwise every finalization writes a
snapshot, even an empty one, before the running entry is removed.
If this closes the design points, #12466 can be reviewed on its own. The
next slice would be S1: the attempt key and deployment identity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
