markhoerth commented on PR #13386: URL: https://github.com/apache/gravitino/pull/13386#issuecomment-5782890431
@roryqi good list, and the doc should state the models explicitly. The choice: The required guarantee is **at-least-once latest-state evaluation**, provided by the scheduler. Coalescing is acceptable for all four activities: maintenance acts on the table's current state, not on the history that produced it. Per-commit evaluation is not required anywhere. The commit path is **best effort by design**. A lost signal delays acceleration, never maintenance, because the next scheduler pass evaluates the table regardless. **Exactly-once job effect** is not achievable and not needed. What is needed is at most one in-flight job per table per activity, and jobs whose repetition is harmless: a repeated compaction wastes work and leaves orphans that orphan cleanup collects. Recovery is driven by the **table's own state**, not the event row: current snapshot id against the id at last measurement, plus per-policy last-run and in-flight job state. An evaluation is owed whenever those disagree, so nothing is replayed. By boundary: - Commit durable before the event, or event durable before the claim: signal lost, evaluated on the next pass. - Node fails holding a claim: the claim needs a lease with an expiry, or the table is stranded until an operator intervenes. On expiry the next pass takes it. - Job accepted before `job_id` is recorded: the only duplicate-submission boundary. Record an idempotency key before submission and reconcile on it. - Deferred for an in-flight job or `minIntervalMs`: a deferral, not a loss. The snapshot ids still disagree. Failure tests follow: kill the node after commit and before the event, mid-evaluation holding a claim, and between submission and `job_id`. In each case the table is evaluated on the next pass and no second job runs concurrently. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
