j1wonpark opened a new issue, #4348: URL: https://github.com/apache/amoro/issues/4348
### What happened? When a table process (e.g. `EXPIRE-SNAPSHOTS`) fails once, AMS resubmits it in a tight infinite loop with no backoff. Each iteration logs the following trio, flooding the log — we observed ~270k occurrences and 1.3GB of logs within 70 minutes, with AMS constantly burning ~1.7 CPU cores: ``` WARN ... Failed to transit status for processEvent: SUBMIT_REQUESTED with initial status: FAILED, process: ... have been marked as terminal status: FAILED. WARN ... Failed to transit status for processEvent: COMPLETE_FAILED with initial status: FAILED, ... WARN ... Failed to transit status for processEvent: RETRY_REQUESTED with initial status: FAILED, ... ``` ### Root cause Since #4116, `ProcessService` constructs `DefaultTableProcessStore` passing `processMeta.getRetryNumber()` (the *current* retry count) into the constructor's `maxRetryTime` parameter (originally `PROCESS_MAX_RETRY_NUMBER` in #3924). For a new process this makes `maxRetryTime = 0`, so on the first failure `isTerminal(FAILED)` evaluates `retryNumber(0) >= maxRetryTime(0)` = true: the process becomes terminal immediately, every subsequent transition is rejected, and `retryNumber` never increases. Meanwhile the retry branch in `executeOrTraceProcess` checks `retryNumber < PROCESS_MAX_RETRY_NUMBER` (0 < 3, always true), ignores the rejected `RETRY_REQUESTED` transition, and unconditionally resubmits — an infinite loop. ### Affects Versions master ### What table formats are you seeing the problem on? _No response_ ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
