j1wonpark opened a new issue, #4295: URL: https://github.com/apache/amoro/issues/4295
## Search before asking - [x] I have searched in the [issues](https://github.com/apache/amoro/issues?q=is%3Aissue) and found no similar issues. ## Description Third and fourth implementation phases of **AIP-5: Dynamic Resource Allocation for Optimizer** (#4191), building on the scale-up machinery of #4271: per-optimizer idle tracking and idle-driven **scale-down with graceful drain** for dynamic-allocation-enabled groups. Combined into one subtask because idle tracking's only consumer is the scale-down decision. Opt-in as before: groups without `dynamic-allocation.enabled = true` are unaffected. - **Idle tracking** is derived from the per-round load snapshot (per-token in-flight counts), not the event-based counter sketched on the AIP page — rationale in the PR; the AIP page will be revised alongside it. - **Scale-down**: one optimizer per round (longest idle past `executor-idle-timeout`), rate-limited by `scale-down-cooldown`, floor-checked against `min-parallelism`, and only in rounds with no active scale-up demand signal. - **Graceful drain**: a draining optimizer stops receiving new tasks and is removed once in-flight tasks finish; a `drain-timeout` force-removal falls back to the existing retry path, so correctness is never affected. - **No new configuration**: consumes `executor-idle-timeout`, `scale-down-cooldown`, and `drain-timeout` (defined since #4253). One new validation rule: `sustained-backlog-timeout ≤ executor-idle-timeout / 2`. Out of scope (final phase): metrics and config docs regeneration. ## Parent issue AIP-5: https://github.com/apache/amoro/issues/4191 ## Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ## Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
