j1wonpark opened a new issue, #4271: URL: https://github.com/apache/amoro/issues/4271
## Search before asking - [x] I have searched in the [issues](https://github.com/apache/amoro/issues?q=is%3Aissue) and found no similar issues. ## Description This is the second implementation phase of **AIP-5: Dynamic Resource Allocation for Optimizer**, building on the configuration foundation of #4253. It adds demand-driven **scale-up** for dynamic-allocation-enabled groups. Scale-down lands in later phases, and everything remains opt-in: groups without `dynamic-allocation.enabled = true` are unaffected. Scope: 1. **`dynamic-allocation.executor-parallelism` (default `1`).** The scaling unit is one homogeneous K-thread optimizer instance (the Spark executor model); this property configures K. Validation: `1 ≤ executor-parallelism ≤ max-parallelism` — a single instance already exceeding the cap could never be created, which would leave an enabled group as a silent no-op. 2. **A dedicated scale keeper.** The existing group keeper checks groups at `optimizer-group.min-parallelism-check-interval × consecutive attempts` (minutes), which would make the DRA backlog timeouts (seconds) unreachable. A new `OptimizerScaleKeeper` (same `AbstractKeeper` infrastructure: HA leader gating, delay queue, lifecycle) evaluates each DRA group at its own `sustained-backlog-timeout`. It owns both the floor and the demand scaling of DRA groups; the legacy keeper skips them but keeps watching, resuming floor duty if DRA is disabled later. Groups are picked up on create, startup load, and update (covering enabling DRA on an existing group at runtime). 3. **Scale-up decision.** Per group and evaluation round: - **Floor**: `min-parallelism` deficits are satisfied immediately (no timing gate), in K-thread instance units. - **Immediate demand**: when `busy + serviceable PLANNED > effective threads`, scale out with an exponential ramp (1, 2, 4, 8 instances) clamped to the actual need — Spark `ExecutorAllocationManager` semantics, including resetting the ramp when the clamp binds or demand clears. - **Future demand**: pending tables while every thread is busy add a single probe instance. This also covers the zero-optimizer cold start: planning is only driven by `pollTask`, so with no optimizer nothing polls, planning never runs, and PLANNED tasks never materialize — pending tables are the only observable demand signal. - Demand must persist for `scheduler-backlog-timeout` before the first scale-out; later rounds are spaced by `sustained-backlog-timeout`, so a trickle drained between rounds never accumulates toward one. The `max-parallelism` cap always wins. 4. **Quota-mode-aware serviceable demand.** A table with a proportional quota (`self-optimizing.quota ≤ 1`) counts its whole PLANNED backlog (scaling raises its limit, which is `ceil(quota × availableCore)`); a table with an absolute quota (`> 1`) counts only free slots, since its fixed limit cannot be raised by scaling. 5. **Boot-window accounting.** Requested-but-unregistered capacity counts toward effective threads, so a booting pod is not re-requested every round. Registration is optimizer-driven and heartbeat expiry only starts after it, so these entries carry their own boot deadline (3 minutes); a request that never registers is pruned instead of freezing scale-up, and a synchronously failed request is retried on a later round. 6. **Planning-bound guard.** Idle threads while tables wait as PENDING and no PLANNED tasks materialize means the bottleneck is the serialized planning (`optimizer.max-planning-parallelism`), not thread capacity — scaling out would only add idle threads. The keeper surfaces this as an edge-triggered warning naming the config to raise, and does not scale. The AIP-5 page will be revised alongside this phase: the scaling-granularity section currently refers to `execution-parallel`, which is not a group-level property, and will document `dynamic-allocation.executor-parallelism` instead. Out of scope for this phase (later phases): per-optimizer idle tracking, scale-down + graceful drain, and the new gauges/counters (the planning-bound condition is log-only here; metrics land in the observability phase). ## Parent issue AIP-5: https://github.com/apache/amoro/issues/4191 ## Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ## Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
