xiangfu0 opened a new pull request, #19962: URL: https://github.com/apache/pinot/pull/19962
The Ratis task ledger can now drive a normal minion starter through scheduling, execution, and fenced offline output publication. A controller commits each finite schedule occurrence and its watermark atomically; an authenticated minion claims the task, generates a real Pinot segment, and commits immutable content plus the publication manifest and `SUCCEEDED` state in one transaction. Stacked on #19961. Helix/ZK remains the default, and native query listeners remain disabled. - Add bounded committed-round scheduling, retry/cancellation recovery, and admission backpressure that preserves capacity for claims and progress. - Add certificate-scoped minion work/command/outcome RPCs, independent heartbeats, durable exact-request retries, and normal `StartMinion` integration. - Fence publication by live worker incarnation, attempt, table incarnation, captured input versions, and immutable output identity. Expired or cancelled workers cannot publish. - Expose authorized experimental table-scoped scheduling/status/cancellation APIs and document the qualification limits. This draft qualifies `NativeSegmentGenerationTask` for digest-pinned local JSON-lines input and an approved shared local output directory. General legacy task/plugin API parity, cron/timeouts, remote deep stores, automatic replica placement, and retention/compaction remain outstanding. Publication records committed content; it does not assert query availability. Validation: - 70 focused tests passed across catalog/snapshot, membership, authenticated transport, controller scheduling, minion execution/outbox, legacy minion configuration/registry, and normal starters. The final affected-behavior rerun passed 27 tests. - The real three-voter, mTLS normal-starter test generated and published a segment, installed it through the existing native assignment protocol, preserved a single publication after minion restart, verified closed query ports, and exercised quorum-loss failure. - Affected-module formatting, Checkstyle, and license checks passed. Compilation used warning/deprecation flags; no warnings matched added Java lines. `git diff --check` passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
