lasdf1234 commented on PR #13532: URL: https://github.com/apache/gravitino/pull/13532#issuecomment-6124797137
@roryqi Agreed — for this use case a per-table single-writer path is the better fit, and we should not add a public merge/PATCH surface just to support one-job-per-spec. Suggested direction: 1. Pin one snapshot per table collection cycle. 2. Call `allManifests` once, group measurements by `partitionSpecId` in that single pass (N specs still produce N aggregates, but not N full-list scans). 3. Publish both complete object-valued statistics with one existing PUT / `updateStatistics` (OCC from #13487 is enough). 4. Enforce per-table single-flight in the optimizer/job layer (and carry snapshot id so an older cycle cannot overwrite a newer one). That keeps ownership clear (one writer for these two statistic names), cuts repeated manifest scanning and Gravitino write pressure, and avoids the Manager-level read/merge/write race you called out. If wall-clock for many specs becomes an issue later, parallelize aggregation **inside** the same table cycle and still finish with one PUT — rather than multiple concurrent writers plus merge. @laserninja Could we rework the collector / #13717 path along these lines and defer (or drop) the merge API from this PR unless we later have a documented need for true multi-writer updates? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
