SEZ9 commented on issue #11364: URL: https://github.com/apache/seatunnel/issues/11364#issuecomment-5842395045
Thanks @CryoThrust — agreed, freshness and value quality should stay independent in the latest-snapshot model, and the ordering identity you describe is the right way to make multi-attempt aggregation deterministic. Concretely, I'd like the contract to read as follows before we freeze it: - `EXACT`, `BEST_EFFORT`, `UNSUPPORTED`, and `UNAVAILABLE` remain per-fact value-quality semantics and are never rewritten by age. An old `EXACT` value is presented as stale via a separate freshness field derived from observation time and the configured freshness policy; its quality stays `EXACT`. - A newer `BEST_EFFORT` or `UNSUPPORTED` report observed inside the window is fresh, even though its quality is lower. - The normalized snapshot retains the winning (vertex, task, attempt, sequence) identity. A lower attempt, or a non-increasing sequence within the same attempt, is treated as an out-of-order update and rejected rather than silently overwriting the current value. - The metrics boundary exposes only bounded freshness/quality aggregates — no positions and no report identities as labels. @goutamadwant, remaining asks before this point is considered settled: 1. Update the STIP text with the four items above so freshness and quality are explicitly distinct fields in the snapshot contract. 2. Add deterministic test coverage for: an old-attempt report arriving after a newer attempt; an out-of-order sequence within one attempt; an old exact value crossing the freshness boundary; and a fresh best-effort/unsupported value. 3. Confirm the ordering identity (attempt + sequence + observation time) is kept visible in the normalized latest snapshot rather than dropped during aggregation. This is a contract clarification only — it should not broaden the current additive latest-snapshot scope, and checkpoint positions stay out of v1. <!-- streview-comment:1322 --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
