wenjin272 commented on PR #1404:
URL: https://github.com/apache/flink-agents/pull/1404#issuecomment-6976211705

   Thanks for working on this and for adding the issue and detailed PR 
description.
   
   Could we reuse the existing `executeAsync` for the retry backoff instead of 
introducing `suspend` and a separate delay mechanism?
   
   Only the wait needs to run through `executeAsync`; the retry loop and the 
durable boundary of each model attempt can stay as they are. Synchronous chat 
would retain its current blocking wait. This would release the mailbox on JDK 
21 and use the existing lock handoff on the older parallel engine, without 
adding another context interface or scheduling mechanism.
   
   The tradeoff is that backoff would occupy a worker. This can delay unrelated 
work sharing the pool, but when requests are being rate-limited by the same 
provider, it also provides some coarse backpressure rather than immediately 
making workers available for more requests that may hit the same limit. It is 
not a substitute for provider-level rate limiting, but I think it is an 
acceptable tradeoff for this focused fix.
   
   I'd prefer starting with this simpler approach. We can revisit worker-free 
backoff if we see thread-pool starvation under mixed workloads. Please add a 
runtime regression test showing that other keys/mailbox work can progress 
during the wait and that retries resume afterwards.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to