purushah opened a new pull request, #1427: URL: https://github.com/apache/flink-agents/pull/1427
Linked issue: #1202 ### Purpose of change A chat request that uses tools resends its whole conversation on every round of the tool loop, so the model input grows. This PR adds a **context management policy** that bounds what the built-in chat action sends, in Java and Python: - **Truncation** (`context-management.tool-results.max-chars`): a longer tool result is cut in the middle. - **Masking** (`tool-results.keep-last-rounds`): results of older rounds become a one-line placeholder. - **Summarization** (`max-input-tokens`): above a share of the budget, older rounds are replaced with a summary written by a chat model. Everything is off by default. The options declare the default policy; a `ChatRequestEvent` or a `ctx.chat` call can carry its own. #### Runtime flow 1. `chat_model_action` asks `ContextManager.prepare` (Python `manager.prepare`) what to send. A request without a tool-call round is returned as it is. 2. The manager takes the request's policy, else the options, and builds the view: summarized rounds are replaced by the summary, older rounds are masked, other oversized results are cut. 3. With a budget it estimates the view (`estimateInputTokens`). Above the trigger it calls the summarizer as a durable call, keeps the summary in sensory memory and rebuilds the view. 4. It emits one `ContextManagementEvent` per step. The action sends the view and stores the original messages. #### Key decisions - **Only the model input changes.** The stored conversation is never rewritten, so the summarizer reads original results. Rejected: rewriting stored messages. - **The round is the unit**: an assistant message with tool calls plus its results. Other messages are never changed. - **A request's policy replaces the default as a whole**, so it can switch a part off. Rejected: merging field by field. - **Estimates are approximate by default**, the same in both languages; a connection can return an exact count. Rejected: a tokenizer dependency. Reported usage only raises an estimate: some providers report cached input tokens separately. - **The summarizer is asked to keep its notes within a quarter of the budget**; notes without a limit grow with every round. ### Behavioral Semantics #### Interaction decisions | Policy | Parts set | Result | |---|---|---| | none | – | Whole conversation sent. No event, no state. | | options | one part | Only that part applies, on every send. | | options | all three | Summarized rounds are dropped, then masking, then truncation. A summary is written only if that view passes the trigger. | | on the request | any | Used for every round of the request; the options are ignored. `disabled()` switches it off. | #### Behavioral contracts 1. Without a policy, or without a round, the model is sent the stored conversation; nothing is read or reported. 2. The stored conversation always holds the full messages. 3. A sent tool result is never longer than `max-chars`, marker included. 4. The last `keep-last-rounds` rounds are sent as they are; with a step of `S` the boundary moves `S` rounds at a time. 5. A summary is written only above `trigger-ratio` of the budget; the last `summarization.keep-last-rounds` rounds are never summarized. 6. A request's policy applies to all its rounds and is removed when the request ends. 7. Each step is reported once as a `ContextManagementEvent`. 8. After a recovery the recorded summary is reused; truncation and masking add no durable call. 9. Java and Python produce the same texts and estimates. 10. With a model router, every candidate is sent the same messages. #### Failure behavior | Failure | Behavior | |---|---| | Invalid option or policy value | Raises, naming the option; job submission fails. | | Summarizer fails, returns no text, is cut off, or is declared in the other language | Absorbed: the request goes on, a warning is logged, the `summarize` event carries `error`, the next model call tries again. | | Chat model cannot estimate its input | Falls back to the approximation. | | Durable execution fails during the summarizer call | Propagates, as for a chat call. | ### Tests Python tests mirror the Java ones. | Contract | Tests (`ChatModelActionContextManagementTest` unless named) | |---|---| | 1 | `withoutAPolicy…`, `aRequestWithoutARound…` | | 2 | `anOversizedResult…StoredInFull…`, `roundsThatArriveWithTheRequestAreStoredInFull` | | 3, 4 | `ContextCompactionTest`, `theResultsOfOlderRoundsAreMasked…`, `maskingInSteps…` | | 5 | `whenTheEstimateExceedsTheTrigger…`, `nothingIsSummarizedWhile…` | | 6 | `aPolicyOnTheRequest…`, `whatIsKeptForARequestIsRemoved…`, e2e tests | | 7 | `truncationMaskingAndSummarizationCombine…` | | 8 | `ContextManagementRecoveryTest`, `aRecordedSummaryIsReused…` | | 9 | `CrossLanguageEventSnapshotTest`, `ContextManagerTest`, `TokenEstimateTest` | | 10 | `everyCandidateOfARoutedRequest…` | | Failures | `aSummaryThatCannotBeWritten…`, `aChatModelDeclaredInPython…`, `CompileUtilsTest` | Not verified: - Real-model runs used OpenAI `gpt-4o-mini` only. - The effect of `mask-step-rounds` on cached tokens was not measured. - Recovery was tested with a restored checkpoint at the operator level, not by killing a cluster job; in Python only at the action level. - A summary's length and content are asked of the summarizer, not checked. <details> <summary>Commands, results and real-model runs</summary> Flink 2.3 and JDK 17 unless noted. | Command | Result | |---|---| | `mvn test`, without the two e2e modules | 4319 run, 0 failures, 117 skipped | | `pytest flink_agents -k "not e2e_tests" -m "not integration"` | 3557 passed, 20 skipped | | `mvn test -pl e2e-test/flink-agents-end-to-end-tests-integration -Pflink-2.3` | 76 run, 0 failures, 21 skipped | | JDK 21 build: `ContextManagementE2ETest`, `ChatCallIntegrationTest`, `ContextManagementRecoveryTest` | 1, 4 and 6 passed | | `pytest flink_agents -k "e2e_tests_integration"` | 86 passed, 18 skipped | | Resource cross-language module, plan compatibility, config parity, resource consistency | pass | | `tools/lint.sh -c`, `tools/check-license.sh`, docs build with Hugo 0.110.0 | pass | The job tests (`ContextManagementE2ETest`, `context_management_test.py`) run one job with a scripted model and a lookup tool, and four requests: the job's policy, a request policy, a disabled policy, and a `ctx.chat` call with a policy. They assert the sink output, the events an action receives and the Trace Log records. Real model: `OpenAiContextManagementExample` (`gpt-4o-mini`) on a standalone Flink 2.3.0 cluster. TaskManager output: ``` [managed] step truncate: 1 tool result(s) [managed] step truncate: 1 tool result(s) [managed] step summarize: 2 round(s) added to notes of 182 characters, estimated input 733 -> 442 tokens [managed] step truncate: 1 tool result(s) [managed] step truncate: 1 tool result(s) [managed] step summarize: 2 round(s) added to notes of 365 characters, estimated input 832 -> 503 tokens [managed] step truncate: 1 tool result(s) [managed] step summarize: 1 round(s) added to notes of 332 characters, estimated input 697 -> 496 tokens [managed] step truncate: 1 tool result(s) [managed] answer: TOTAL=210 | input tokens of the last model call: 493 [unmanaged] answer: TOTAL=210 | input tokens of the last model call: 5728 ``` Repeated 26 times, every run answered `TOTAL=210`. The same agent as a Python job answered `TOTAL=210` with 518 input tokens in the last call. With a request policy that only masks (`keep-last-rounds: 2`) the last call used 705 input tokens and the answer was `TOTAL=80`: the model cannot read masked results again, which the docs warn about. Implementation invariants: - Per request, sensory memory keeps under `_CONTEXT_MANAGEMENT_CONTEXT` the request's policy, the summary, the number of summarized rounds and the estimate correction factor. It is removed with the request's other state. - The summarizer is called without the chat model's prompt and tools, as a durable call with the id `summarize`, and follows `chat.async`. It is shown the instructions and user messages the chat model is sent, marked as context, and is asked for facts from the shown rounds only. - Events of a `ctx.chat` call stay inside the call; the Trace Log records them as `_internal_subgraph` records with `delegate_type: _context_management_event`. - The placeholder, marker and summarizer texts and the approximate estimate are pinned by mirrored tests in both languages. </details> ### API Additive only. - New: `ContextManagementPolicy`, `ContextManagementEvent`, `TokenEstimate`, `estimateInputTokens` on the chat model connection and setup, `ChatRequestEvent.withContextManagement` (Python `context_management`, also on `ctx.chat`), seven `context-management.*` options. - Unchanged: without a policy the chat action sends what it sent before; a `ChatRequestEvent` without the new attribute serializes as before. ### Documentation - [ ] `doc-needed` - [ ] `doc-not-needed` - [x] `doc-included` ### Was this patch authored or co-authored using generative AI tooling? - [x] Yes - [ ] No Generated-by: Claude Code 2.1.329 (Claude Fable 5.2) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
