purushah opened a new pull request, #1427:
URL: https://github.com/apache/flink-agents/pull/1427

   Linked issue: #1202
   
   ### Purpose of change
   
   A chat request that uses tools resends its whole conversation on every round 
of the tool loop, so the model input grows. This PR adds a **context management 
policy** that bounds what the built-in chat action sends, in Java and Python:
   
   - **Truncation** (`context-management.tool-results.max-chars`): a longer 
tool result is cut in the middle.
   - **Masking** (`tool-results.keep-last-rounds`): results of older rounds 
become a one-line placeholder.
   - **Summarization** (`max-input-tokens`): above a share of the budget, older 
rounds are replaced with a summary written by a chat model.
   
   Everything is off by default. The options declare the default policy; a 
`ChatRequestEvent` or a `ctx.chat` call can carry its own.
   
   #### Runtime flow
   
   1. `chat_model_action` asks `ContextManager.prepare` (Python 
`manager.prepare`) what to send. A request without a tool-call round is 
returned as it is.
   2. The manager takes the request's policy, else the options, and builds the 
view: summarized rounds are replaced by the summary, older rounds are masked, 
other oversized results are cut.
   3. With a budget it estimates the view (`estimateInputTokens`). Above the 
trigger it calls the summarizer as a durable call, keeps the summary in sensory 
memory and rebuilds the view.
   4. It emits one `ContextManagementEvent` per step. The action sends the view 
and stores the original messages.
   
   #### Key decisions
   
   - **Only the model input changes.** The stored conversation is never 
rewritten, so the summarizer reads original results. Rejected: rewriting stored 
messages.
   - **The round is the unit**: an assistant message with tool calls plus its 
results. Other messages are never changed.
   - **A request's policy replaces the default as a whole**, so it can switch a 
part off. Rejected: merging field by field.
   - **Estimates are approximate by default**, the same in both languages; a 
connection can return an exact count. Rejected: a tokenizer dependency. 
Reported usage only raises an estimate: some providers report cached input 
tokens separately.
   - **The summarizer is asked to keep its notes within a quarter of the 
budget**; notes without a limit grow with every round.
   
   ### Behavioral Semantics
   
   #### Interaction decisions
   
   | Policy | Parts set | Result |
   |---|---|---|
   | none | – | Whole conversation sent. No event, no state. |
   | options | one part | Only that part applies, on every send. |
   | options | all three | Summarized rounds are dropped, then masking, then 
truncation. A summary is written only if that view passes the trigger. |
   | on the request | any | Used for every round of the request; the options 
are ignored. `disabled()` switches it off. |
   
   #### Behavioral contracts
   
   1. Without a policy, or without a round, the model is sent the stored 
conversation; nothing is read or reported.
   2. The stored conversation always holds the full messages.
   3. A sent tool result is never longer than `max-chars`, marker included.
   4. The last `keep-last-rounds` rounds are sent as they are; with a step of 
`S` the boundary moves `S` rounds at a time.
   5. A summary is written only above `trigger-ratio` of the budget; the last 
`summarization.keep-last-rounds` rounds are never summarized.
   6. A request's policy applies to all its rounds and is removed when the 
request ends.
   7. Each step is reported once as a `ContextManagementEvent`.
   8. After a recovery the recorded summary is reused; truncation and masking 
add no durable call.
   9. Java and Python produce the same texts and estimates.
   10. With a model router, every candidate is sent the same messages.
   
   #### Failure behavior
   
   | Failure | Behavior |
   |---|---|
   | Invalid option or policy value | Raises, naming the option; job submission 
fails. |
   | Summarizer fails, returns no text, is cut off, or is declared in the other 
language | Absorbed: the request goes on, a warning is logged, the `summarize` 
event carries `error`, the next model call tries again. |
   | Chat model cannot estimate its input | Falls back to the approximation. |
   | Durable execution fails during the summarizer call | Propagates, as for a 
chat call. |
   
   ### Tests
   
   Python tests mirror the Java ones.
   
   | Contract | Tests (`ChatModelActionContextManagementTest` unless named) |
   |---|---|
   | 1 | `withoutAPolicy…`, `aRequestWithoutARound…` |
   | 2 | `anOversizedResult…StoredInFull…`, 
`roundsThatArriveWithTheRequestAreStoredInFull` |
   | 3, 4 | `ContextCompactionTest`, `theResultsOfOlderRoundsAreMasked…`, 
`maskingInSteps…` |
   | 5 | `whenTheEstimateExceedsTheTrigger…`, `nothingIsSummarizedWhile…` |
   | 6 | `aPolicyOnTheRequest…`, `whatIsKeptForARequestIsRemoved…`, e2e tests |
   | 7 | `truncationMaskingAndSummarizationCombine…` |
   | 8 | `ContextManagementRecoveryTest`, `aRecordedSummaryIsReused…` |
   | 9 | `CrossLanguageEventSnapshotTest`, `ContextManagerTest`, 
`TokenEstimateTest` |
   | 10 | `everyCandidateOfARoutedRequest…` |
   | Failures | `aSummaryThatCannotBeWritten…`, `aChatModelDeclaredInPython…`, 
`CompileUtilsTest` |
   
   Not verified:
   - Real-model runs used OpenAI `gpt-4o-mini` only.
   - The effect of `mask-step-rounds` on cached tokens was not measured.
   - Recovery was tested with a restored checkpoint at the operator level, not 
by killing a cluster job; in Python only at the action level.
   - A summary's length and content are asked of the summarizer, not checked.
   
   <details>
   <summary>Commands, results and real-model runs</summary>
   
   Flink 2.3 and JDK 17 unless noted.
   
   | Command | Result |
   |---|---|
   | `mvn test`, without the two e2e modules | 4319 run, 0 failures, 117 
skipped |
   | `pytest flink_agents -k "not e2e_tests" -m "not integration"` | 3557 
passed, 20 skipped |
   | `mvn test -pl e2e-test/flink-agents-end-to-end-tests-integration 
-Pflink-2.3` | 76 run, 0 failures, 21 skipped |
   | JDK 21 build: `ContextManagementE2ETest`, `ChatCallIntegrationTest`, 
`ContextManagementRecoveryTest` | 1, 4 and 6 passed |
   | `pytest flink_agents -k "e2e_tests_integration"` | 86 passed, 18 skipped |
   | Resource cross-language module, plan compatibility, config parity, 
resource consistency | pass |
   | `tools/lint.sh -c`, `tools/check-license.sh`, docs build with Hugo 0.110.0 
| pass |
   
   The job tests (`ContextManagementE2ETest`, `context_management_test.py`) run 
one job with a scripted model and a lookup tool, and four requests: the job's 
policy, a request policy, a disabled policy, and a `ctx.chat` call with a 
policy. They assert the sink output, the events an action receives and the 
Trace Log records.
   
   Real model: `OpenAiContextManagementExample` (`gpt-4o-mini`) on a standalone 
Flink 2.3.0 cluster. TaskManager output:
   
   ```
   [managed] step truncate: 1 tool result(s)
   [managed] step truncate: 1 tool result(s)
   [managed] step summarize: 2 round(s) added to notes of 182 characters, 
estimated input 733 -> 442 tokens
   [managed] step truncate: 1 tool result(s)
   [managed] step truncate: 1 tool result(s)
   [managed] step summarize: 2 round(s) added to notes of 365 characters, 
estimated input 832 -> 503 tokens
   [managed] step truncate: 1 tool result(s)
   [managed] step summarize: 1 round(s) added to notes of 332 characters, 
estimated input 697 -> 496 tokens
   [managed] step truncate: 1 tool result(s)
   [managed] answer: TOTAL=210 | input tokens of the last model call: 493
   [unmanaged] answer: TOTAL=210 | input tokens of the last model call: 5728
   ```
   
   Repeated 26 times, every run answered `TOTAL=210`. The same agent as a 
Python job answered `TOTAL=210` with 518 input tokens in the last call. With a 
request policy that only masks (`keep-last-rounds: 2`) the last call used 705 
input tokens and the answer was `TOTAL=80`: the model cannot read masked 
results again, which the docs warn about.
   
   Implementation invariants:
   - Per request, sensory memory keeps under `_CONTEXT_MANAGEMENT_CONTEXT` the 
request's policy, the summary, the number of summarized rounds and the estimate 
correction factor. It is removed with the request's other state.
   - The summarizer is called without the chat model's prompt and tools, as a 
durable call with the id `summarize`, and follows `chat.async`. It is shown the 
instructions and user messages the chat model is sent, marked as context, and 
is asked for facts from the shown rounds only.
   - Events of a `ctx.chat` call stay inside the call; the Trace Log records 
them as `_internal_subgraph` records with `delegate_type: 
_context_management_event`.
   - The placeholder, marker and summarizer texts and the approximate estimate 
are pinned by mirrored tests in both languages.
   
   </details>
   
   ### API
   
   Additive only.
   
   - New: `ContextManagementPolicy`, `ContextManagementEvent`, `TokenEstimate`, 
`estimateInputTokens` on the chat model connection and setup, 
`ChatRequestEvent.withContextManagement` (Python `context_management`, also on 
`ctx.chat`), seven `context-management.*` options.
   - Unchanged: without a policy the chat action sends what it sent before; a 
`ChatRequestEvent` without the new attribute serializes as before.
   
   ### Documentation
   
   - [ ] `doc-needed`
   - [ ] `doc-not-needed`
   - [x] `doc-included`
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   - [x] Yes
   - [ ] No
   
   Generated-by: Claude Code 2.1.329 (Claude Fable 5.2)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to