wenjin272 opened a new issue, #1279: URL: https://github.com/apache/flink-agents/issues/1279
### Search before asking - [x] I searched in the [issues](https://github.com/apache/flink-agents/issues) and found no issue tracking this failure path. ### Description When a model-driven sub-agent call raises while its returned handle is being awaited, `ToolCallAction` catches the exception and creates a failed tool observation. However, the handle remains registered as pending. The action's finalization hook then throws `finished without resolving the sub-agent calls`, failing the action instead of allowing the model to receive and react to the failed delegation. This is a mismatch between the model-driven dispatcher's error handling and the runtime's handle lifecycle. The [Sub-Agents guide](https://github.com/apache/flink-agents/blob/b5fc727927bc4e35301e58278b682efe894d25e8/docs/content/docs/development/multi-agent/subagents.md#L1185-L1200) explicitly says that exceptions while submitting or awaiting a model-driven delegation are reported as tool observations. #### Trigger conditions 1. The built-in `ToolCallAction` dispatches a model-requested sub-agent call. 2. `submit()` succeeds and returns a handle registered in the executing action's pending-call registry. 3. Awaiting that handle raises before it is marked completed/consumed. For example, an external deferred callable raises, or an async setup's status/fetch operation raises. 4. The dispatcher converts that exception into a failed observation and reaches action finalization with the handle still pending. A single call in the sequential path is sufficient. Parallel execution, failover, and a cyclic result are not required. Returning `SubagentResult.error(...)` normally is unaffected: the await succeeds with an error value and consumes the handle. **Expected:** If the model-driven dispatcher handles an exception by producing a failed observation, the corresponding handle is also finalized, and action completion does not fail the unresolved-handle check. **Actual:** A failed observation is constructed, but action finalization fails with: ```text Action tool_call_action finished without resolving the sub-agent calls it submitted: [<session-id>#<call-id>]. Resolve every handle returned by submit(), individually or through SubagentFutures. ``` This does not request suppressing system-level exceptions from caller-driven actions. Those exceptions are documented to fail the action and trigger recovery; that behavior should remain intact. ### How to reproduce #### Minimal Python reproduction From a checkout of the affected revision, activate the repository's Python development environment with its test dependencies installed. Save the following as `reproduce_pending_handle.py` in the repository root. The script reuses existing repository test fixtures. It runs the real `process_tool_request`, deferred setup, future, registry, and lifecycle hook. Only durable execution is replaced with an in-process awaitable that executes the supplied callable, so no model service or Flink cluster is needed. ```python import asyncio from flink_agents.api.subagent import CALLABLE_NAME_PREFIX from flink_agents.plan.actions.tool_call_action import process_tool_request from flink_agents.plan.tests.actions.test_tool_call_action_subagent import ( _Context, _request, ) from flink_agents.runtime.deferred_subagent import DeferredSubagentSetup from flink_agents.runtime.tests.test_base_subagent import _FakeTask class Context(_Context): def durable_execute_async(self, func, *args, **kwargs): kwargs.pop("durable_id", None) kwargs.pop("reconciler", None) async def execute(): return func(*args, **kwargs) return execute() class FailingReviewer(DeferredSubagentSetup): def prepare(self, ctx, prompt, session_id, call_id): def call(): raise RuntimeError("remote unavailable") return (f"{session_id}#{call_id}", call, None) setup = FailingReviewer() setup.set_subagent_name("reviewer") task = _FakeTask(action_name="tool_call_action") setup.on_action_prepared(task) ctx = Context().with_agent("reviewer", setup) request = _request( "model", [{ "id": "call-0", "function": { "name": CALLABLE_NAME_PREFIX + "reviewer", "arguments": {"code": "hello"}, }, }], ) asyncio.run(process_tool_request(request, ctx)) response = ctx.sent_events[0] assert response.success["call-0"] is False assert "remote unavailable" in response.error["call-0"] print("The dispatcher returned a failed tool observation.") print("Pending registry empty:", setup.pending_call_registry().is_empty()) # Expected to finish normally after the dispatcher handled the exception. # Actual: RuntimeError: Action tool_call_action finished without resolving ... setup.on_action_finishing(task) ``` Run from the repository root: ```bash export PYTHONPATH="$PWD/python:$(python -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])')" python reproduce_pending_handle.py ``` Observed output: ```text The dispatcher returned a failed tool observation. Pending registry empty: False ... File ".../runtime/base_subagent.py", line 210, in on_action_finishing registry.check_empty() File ".../runtime/subagent_handles.py", line 72, in check_empty raise RuntimeError(msg) RuntimeError: Action tool_call_action finished without resolving the sub-agent calls it submitted: [...] ``` The same lifecycle reproduction also fails for an async setup whose `call_query_status()` raises, and for both deferred and async setups with parallel model tool-call dispatch enabled: all four combinations construct a failed observation and then fail finalization. #### Java operator confirmation The behavior was also reproduced with the actual `ActionExecutionOperator` using the harness in `InternalSubagentCallTest`: 1. Register an external `BaseDeferredSubagentSetup` as the `child` AGENT resource. 2. Use the existing `requestChildTool` / `receiveChildTool` helpers to emit and consume a `_subagent_child` request. Remove the built-in `chat_model_action` from the test plan so this isolated test handles the tool response itself. 3. In the setup's durable callable, return `SubagentResult.ok(payload)` where `payload` is a map containing itself. This makes Java durable-result serialization fail before tool-result normalization. It is a supplemental trigger for the same await-failure path, not a prerequisite for the bug. 4. Run the plan through the existing `run(...)` operator-harness helper. Instead of completing with an error observation, the operator fails with: ```text Caused by: java.lang.IllegalStateException: Action tool_call_action finished without resolving the sub-agent calls it submitted: [...] at PendingSubagentCallRegistry.checkEmpty(PendingSubagentCallRegistry.java:58) at BaseSubagentSetup.onActionFinishing(BaseSubagentSetup.java:87) at ActionExecutionOperator.notifyActionFinishing(ActionExecutionOperator.java:1386) at ActionExecutionOperator.processActionTask(ActionExecutionOperator.java:917) ``` ### Root cause The success and exception paths disagree on who owns handle completion: 1. **Handles unregister only after a successful await.** Python [DeferredSubagentFuture](https://github.com/apache/flink-agents/blob/b5fc727927bc4e35301e58278b682efe894d25e8/python/flink_agents/runtime/deferred_subagent.py#L89-L124) calls `_resolve` after durable execution returns; [AsyncSubagentFuture](https://github.com/apache/flink-agents/blob/b5fc727927bc4e35301e58278b682efe894d25e8/python/flink_agents/runtime/async_subagent.py#L145-L158) marks the handle consumed and unregisters it only after the await returns. The equivalent Java paths are [DeferredSubagentFuture](https://github.com/apache/flink-agents/blob/b5fc727927bc4e35301e58278b682efe894d25e8/runtime/src/main/java/org/apache/flink/agents/runtime/subagent/DeferredSubagentFuture.java#L77-L106) and [AsyncSubagentFuture](https://github.com/apache/flink-agents/blob/b5fc727927bc4e35301e58278b682efe894d25e8/runtime/src/main/java/org/apache/flink/agents/runtime/subagent/AsyncSubagentFuture.java#L81-L95). An exception skips thes e steps. 2. **The model-driven dispatcher handles the exception without finalizing the handle.** Python [sequential and parallel dispatch](https://github.com/apache/flink-agents/blob/b5fc727927bc4e35301e58278b682efe894d25e8/python/flink_agents/plan/actions/tool_call_action.py#L425-L484) records a failed observation in its exception handlers. Java does the same in [ToolCallAction](https://github.com/apache/flink-agents/blob/b5fc727927bc4e35301e58278b682efe894d25e8/plan/src/main/java/org/apache/flink/agents/plan/actions/ToolCallAction.java#L445-L546). 3. **Action finalization rejects the leftover entry.** [BaseSubagentSetup.onActionFinishing](https://github.com/apache/flink-agents/blob/b5fc727927bc4e35301e58278b682efe894d25e8/runtime/src/main/java/org/apache/flink/agents/runtime/subagent/BaseSubagentSetup.java#L81-L88) checks that the registry is empty. The [operator invokes this hook](https://github.com/apache/flink-agents/blob/b5fc727927bc4e35301e58278b682efe894d25e8/runtime/src/main/java/org/apache/flink/agents/runtime/operator/ActionExecutionOperator.java#L911-L921) before recording successful action completion. Python has the corresponding check in [on_action_finishing](https://github.com/apache/flink-agents/blob/b5fc727927bc4e35301e58278b682efe894d25e8/python/flink_agents/runtime/base_subagent.py#L205-L210). The fix needs to align handled model-delegation failures with handle finalization, while preserving intentional cancellation and caller-driven system-failure behavior. Regression tests should include the actual lifecycle check, cover deferred/async and sequential/parallel dispatch, and retain normal `SubagentResult.error(...)` as a control. Testing only the returned tool observation misses the failure that occurs when the action finishes. ### Version and environment - Apache Flink Agents `0.4-SNAPSHOT`. - Reproduced at `b5fc727927bc4e35301e58278b682efe894d25e8`; the relevant production code and reproduction fixtures are unchanged at current main `bc62aa89d0a3cb2d5ce1a5bd9598df9086e35f12`. - Python 3.11; Java operator verification with JDK 21.0.8, compiled from the tested source revision; macOS arm64. - Python verification uses the real dispatch/lifecycle with an in-process durable-execution test double. Java verification uses the actual operator harness. No live LLM or external agent service was used. ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
