lasdf1234 commented on PR #13386: URL: https://github.com/apache/gravitino/pull/13386#issuecomment-5777510994
@markhoerth Hi Mark, On triggers and orchestration, here is what we are proposing based on industry practice (Floe, IOMETE, Glue, Amoro) and how we want to shape the TMS design. Please confirm before we update the doc. 1. Trigger model Industry practice ties write/commit acceleration mainly to compaction. Manifest rewrite, snapshot expiration, and orphan cleanup are scheduler-driven (metrics and/or time), because they must run even when a table stops committing, and orphan files often come from failed writes with no commit event. We propose: Commit path = compaction accelerator only (dirty flag and/or direct compaction evaluate — whichever you prefer). Scheduler = main clock for all four types. Manifest rewrite, snapshot expiration, and orphan cleanup are not submitted directly from the commit hook. 2. Per-table pipeline (scheduler orchestration) There is no single industry-standard per-table pipeline. Floe and IOMETE schedule and evaluate each operation independently; orchestration products like LakeOps chain four steps but in a different order (expire → orphans → compact → manifests). We still want a scheduler-orchestrated ordered pass per table: compaction → manifest rewrite → snapshot expiration Each step is gated by its effective policy, Recommender, and minIntervalMs. We treat this as an explicit product choice to reconcile “four separate policies” with “fixed in-table order,” not as something every vendor does the same way. 3. Orphan cleanup — outside the hot pipeline Orphan cleanup would sit outside the per-table hot pipeline above, as a separate scheduler track with a much longer cadence (industry default ~weekly — e.g. Floe P7D, IOMETE weekly cron), plus an olderThan retention window (typically 3–7 days). Rationale: full storage scan, no cheap metadata signal to evaluate (per IOMETE), and low frequency. Compaction may be reachable from both commit and scheduler; we dedupe with minIntervalMs and in-flight job state. Does this align with your intent — especially whether orphan should remain step 4 in the same pass, or run as a separate weekly scheduler track? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
