wenjin272 commented on code in PR #1208:
URL: https://github.com/apache/flink-agents/pull/1208#discussion_r4236799047
##########
docs/content/docs/operations/configuration.md:
##########
@@ -134,10 +134,10 @@ Here is the list of all built-in core configuration
options.
| `retry-wait-interval` | 1 | int
| Base wait interval in seconds between retries. Uses exponential backoff:
the actual wait time for the Nth retry is `retry-wait-interval * 2^(N-1)`
seconds. For example, with default 1s, waits are 1s, 2s, 4s, etc. Retry count
and total wait time are reported in `ChatResponseEvent` and recorded as metrics
(`retryCount`, `retryWaitSec`) under the configured ChatModel resource name. |
| `chat.async` | true | boolean
| Whether chat asynchronously for built-in chat action.
|
| `tool-call.async` | true | boolean
| Whether the built-in tool-call action runs each tool via durable async
execution.
|
-| `tool-call.parallelism` | os cpu count | int
| In-flight concurrency for tool calls from one `ToolRequestEvent` batch when
`tool-call.async` is enabled. `1` runs tools serially; values greater than `1`
run a parallel durable batch with a sliding window of at most that many
concurrent tool calls. On **Java**, concurrent in-batch execution requires
**JDK 21+** (Continuation API); below JDK 21 the batch still runs but tool
calls execute serially. **Python** uses the shared async `ThreadPoolExecutor`
and runs batches concurrently regardless of JDK version. Increases in-flight
external calls; after failover, unfinished tools may be submitted again —
side-effecting tools should be idempotent or provide a reconciler. {{< hint
warning >}}**Default is parallel** (`os cpu count`). Chat, RAG, and tool
batches share one `num-async-threads` pool **per operator subtask** (all keys
on that subtask). Built-in actions for a single key run one at a time, so chat
and a tool batch on the **same key** do not overlap in the usual chat → tool
path; delay shows up mainly **across keys** on the same subtask. With defaults
(`num-async-threads = 2× cores`, `tool-call.parallelism = cores`), one batch
can use up to half the pool; several busy keys can still saturate it. Lower
this value or increase `num-async-threads` on hot subtasks. {{< /hint >}} |
-| `tool-call.batch.timeout.ms` | -1 (disabled) | long
(milliseconds) | Overall timeout for one parallel tool-call batch.
Non-positive disables it. On timeout, completed slots keep their outcome; slots
that started but did not finish are recorded as failures; slots that never
started executing (for example, queued in a saturated pool) stay pending, so
they are re-executed after recovery instead of recording a false failure.
Timeout cancellation is best-effort; external side effects from unfinished tool
calls may still complete, so side-effecting tools should be idempotent or
provide a reconciler. On **Java**, only enforced on **JDK 21+**; on JDK 11 the
batch fallback ignores this setting and runs to completion serially. **Python**
enforces the deadline in the batch await loop. {{< hint warning >}}**Thread
reclamation:** a timeout unblocks the batch but cannot interrupt a tool that is
still running; its worker thread stays in the shared `num-async-threads` pool
until th
e tool returns on its own, so a tool that never returns permanently reduces
pool capacity. Bound blocking work inside the tool (for example an HTTP client
read timeout) rather than relying on this timeout to free the thread.{{< /hint
>}} |
+| `async.batch.parallelism` | os cpu count | int | Maximum concurrent calls in
one `ctx.gather(...)` batch, including ordinary calls, durable calls, and
built-in async tool calls. `1` runs calls sequentially. Actual concurrency is
also limited by `num-async-threads`; lower this per-batch limit to leave
capacity for other calls on the same subtask. Java and Python support batch
concurrency on all supported JDK versions. |
Review Comment:
Thanks for raising this. The rename is intentional: we do not guarantee
backward compatibility across 0.x releases, and these two options were only
recently added and have not appeared in a release yet. I think we can rename
them directly without deprecated aliases or a migration note.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
