wenjin272 opened a new pull request, #1163: URL: https://github.com/apache/flink-agents/pull/1163
Linked issue: #1059 (follow-up to #1060; the tracking issue remains open) ### Purpose of change Preserve multimodal responses during Java structured-output parsing, prevent raw media from appearing in the Python chat failure debug message, and align Java/Python media field validation. #### Runtime flow Java structured-output parsing reads the text projection, parses it into the requested schema, and returns a message preserving the original blocks and tool calls with the parsed value added to `extraArgs`. Python's terminal FAIL path logs the request ID and message count, then rethrows the original exception. Media construction/deserialization validates field types before messages enter normal JSON, bridge, or state paths. #### Key decisions - Use field-local Jackson deserializers to match Python strict media fields without changing unrelated ObjectMapper coercion rules. - Keep Python `blocks` as a list, with construction and replacement-assignment validation. In-place mutations remain normal Python list operations. - Treat Base64 as an opaque non-empty string: no payload scan or decoding at construction. Inferred size assumes valid standard Base64 and is clamped to zero for malformed short input. - Hide raw source values in diagnostic representations while keeping normal serialization lossless. ### Behavioral Semantics #### Interaction decisions | Conditions | Result | |---|---| | Structured output + text/media blocks | Parse the text projection; preserve all original blocks, text formatting, tool calls, and existing metadata. | | Python chat failure + FAIL strategy | Log request ID/count without rendering input messages; propagate the exception. IGNORE/RETRY behavior is unchanged. | | Media + ordinary JSON or bridge maps | Preserve payload/URL; apply the same strict media-field rules during deserialization. | | Media + diagnostic representation | Python source repr hides payload/URL; Java/Python URL string representations are redacted. | | Missing blocks vs explicit null | Missing defaults to empty; explicit null or null elements fail at validated entry points. | | Python field replacement vs list mutation | Replacement is validated; append is supported and does not trigger Pydantic validation. | #### Behavioral contracts 1. Java structured-output parsing adds the parsed result without discarding the original message content or tool calls. 2. The Python FAIL debug message does not include input message bodies; source representations omit raw Base64/URLs. 3. Media string fields reject numeric/boolean coercion. Optional `size_bytes` accepts only integers in `[0, 2^63 - 1]` or null. 4. Java constructors/setters/map conversion reject null block containers/elements. Python construction and field replacement reject them; construction copies the input list and valid append remains supported. 5. Python prompt maps may omit blocks, but explicit null is rejected. 6. Base64 contents remain unchanged through serialization, with no encoding validation. Inferred sizes are non-negative and exact for valid standard Base64 with optional padding. #### Failure behavior Invalid media fields and block collections raise construction/deserialization errors rather than being coerced or dropped. Failed block replacement leaves existing blocks unchanged. Python `validate_assignment` applies to every ChatMessage field, not only blocks. Structured-output parse errors and chat retry policy are unchanged. This is not blanket log sanitization: arbitrary exception text and generic maps are outside these guarantees. Restoring Python-originated built-in Event types remains separately tracked in #1125. ### Tests | Contract | Regression coverage | |---|---| | 1: Preserve structured output content | Java `structuredOutputPreservesOriginalBlocks`; Python `test_structured_output_preserves_original_blocks` | | 2: Safe diagnostic output | Python chat FAIL/caplog regression and `test_media_representations_hide_payloads_but_wire_preserves_them`; Java `base64PayloadPreservationAndSafeRepresentations` | | 3: Strict media fields | Java `mediaFieldsRejectScalarCoercionOnJsonAndMapPaths`; Python media-string/source/size parameterized tests | | 4: Block collection boundaries | Java `nullBlocksFailAtEveryEntryPoint`; Python null construction/assignment and list-copy/append/dump tests | | 5: Prompt missing/null distinction | Java `testOmittedBlocksDefaultToEmptyButExplicitNullIsRejected` | | 6: Opaque Base64 and inferred size | Java payload-preservation/size test; Python `test_base64_preserves_unvalidated_payload` and `test_base64_size_and_wire_preservation` | Coverage focuses on wire compatibility, map conversion, accidental media disclosure, and content preservation. Python API/Plan/runtime-tests regression: 1046 passed, 13 skipped. Final Java targeted regression: 122 tests, 11 skipped, zero failures/errors, covering message JSON, structured output, prompt maps, resource adapter, Event Log, and state serde. Not verified: live provider calls, full E2E jobs, skipped cross-language snapshot cases, non-empty tool-call preservation during structured-output parsing, exhaustive assignment validation for non-block fields, or generic-map Event Log sanitization. No provider converter or Event-type restoration is implemented here. <details> <summary>Verification commands and implementation details</summary> - `mvn test -pl runtime -am -Dtest=ChatMessageSerializationTest,ChatMessageTest,ChatModelActionTest,PythonPromptTest,JavaResourceAdapterTest,FileEventLoggerTest,ActionStateSerdeTest,CrossLanguageEventSnapshotTest -Dsurefire.failIfNoSpecifiedTests=false` (JDK 17; rebuilds changed sources). - `PYTHONPATH=<venv-site-packages> python/.venv/bin/pytest -q python/flink_agents/api python/flink_agents/plan python/flink_agents/runtime/tests -m 'not integration'`. - Changed Python files pass Ruff; Java changes are Spotless-formatted; `git diff --check` passes. - Java block lists remain immutable snapshots. Bridge construction initializes an empty list before applying validated map blocks. Python JSON and `model_dump()` continue emitting lists. </details> ### API Wire shape is unchanged. Inputs relying on scalar coercion, invalid sizes, or Java null block lists are now rejected; Python field reassignment is now validated. Python list append remains available. Java structured output retains original text formatting instead of replacing it with cleaned JSON text. URL diagnostic strings become redacted. Normal JSON/state/bridge serialization retains the original media data and URL. ### Documentation - [ ] `doc-needed` - [ ] `doc-not-needed` - [x] `doc-included` ### Was this patch authored or co-authored using generative AI tooling? - [x] Yes - [ ] No Generated-by: Codex (model/version not exposed) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
