wenjin272 commented on PR #1404: URL: https://github.com/apache/flink-agents/pull/1404#issuecomment-6976211705
Thanks for working on this and for adding the issue and detailed PR description. Could we reuse the existing `executeAsync` for the retry backoff instead of introducing `suspend` and a separate delay mechanism? Only the wait needs to run through `executeAsync`; the retry loop and the durable boundary of each model attempt can stay as they are. Synchronous chat would retain its current blocking wait. This would release the mailbox on JDK 21 and use the existing lock handoff on the older parallel engine, without adding another context interface or scheduling mechanism. The tradeoff is that backoff would occupy a worker. This can delay unrelated work sharing the pool, but when requests are being rate-limited by the same provider, it also provides some coarse backpressure rather than immediately making workers available for more requests that may hit the same limit. It is not a substitute for provider-level rate limiting, but I think it is an acceptable tradeoff for this focused fix. I'd prefer starting with this simpler approach. We can revisit worker-free backoff if we see thread-pool starvation under mixed workloads. Please add a runtime regression test showing that other keys/mailbox work can progress during the wait and that retries resume afterwards. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
