wenjin272 opened a new pull request, #1207: URL: https://github.com/apache/flink-agents/pull/1207
Linked issue: #1195 Fixes #1195 ### Purpose of change Callers can construct image, audio, video and document blocks directly from raw bytes, without repeating Base64 encoding after file reads or media generation. #### Runtime flow Python `from_bytes(media_type, data, **kwargs)` validates the byte input, encodes it and delegates to `from_base64`. Java `fromBytes(mediaType, byte[])` uses a shared package-private encoder and the existing block constructor. Both produce the existing `Base64Source`, so serialization and provider conversion follow the established paths. #### Key decisions Encode at construction using standard Base64 without line wrapping. This preserves the current wire format and avoids retaining Java's mutable input array. Python accepts `bytes` explicitly rather than coercing strings, lists or other buffer types. Keep MIME types explicit and keep existing metadata conventions. ### Behavioral Semantics #### Interaction decisions | Entry point / input | Result | |---|---| | Bytes factory, non-empty bytes and valid media fields | Encode once and construct the requested block type. | | Bytes factory, null/non-bytes or empty data | Fail immediately; no block is returned. | | Bytes factory, valid bytes but invalid media fields | Existing constructor validation rejects the block. | | Base64 factory, non-empty string | Preserve the string, including the existing absence of Base64 syntax validation. | | URL factory | Continue referencing externally managed media. | #### Behavioral contracts 1. All four block types preserve arbitrary binary bytes, including non-UTF-8 values, with standard Base64 padding and no line wrapping. 2. Invalid byte inputs and invalid media fields are rejected. Python's optional metadata is passed through using existing validation; Java's two-argument factories leave optional metadata unset. 3. Mutating a Java input array after construction cannot change the block. 4. Bytes-created blocks retain the existing serialization shape and can be read across Java/Python; representative provider conversion preserves their payload. Unsupported provider modalities remain unsupported. 5. String representations and existing sanitized logging projections do not expose media payloads. 6. Existing Base64 factories preserve supplied strings without re-encoding or introducing stricter syntax validation. #### Failure behavior Python raises `TypeError` for non-`bytes` data (including `None`, `bytearray` and `memoryview`), `ValueError` for empty bytes, and existing Pydantic validation errors for invalid media fields. Java raises `IllegalArgumentException` for null/empty data or media types. Construction performs no I/O, retries or fallback; it does not inspect media contents to verify the declared MIME type. Provider capability errors are unchanged. ### Tests | Contract | Coverage | |---|---| | 1. Binary preservation, padding and line wrapping | Python `test_from_bytes_preserves_binary_and_wire_contract`; Java `preservesBinaryPayloadAndExistingWireShape`, parameterized over all four factories. | | 2. Input validation and metadata | Python `test_from_bytes_rejects_non_bytes`, `test_from_bytes_validation_and_metadata`; Java `rejectsMissingDataAndMediaType` and unset-metadata assertions. | | 3. Java array isolation | `doesNotRetainCallerArray`. | | 4. Serialization, language boundaries and provider conversion | New raw-media snapshots generated independently in both languages, stability checks and opposite-language deserialization; Python `test_raw_bytes_preserved_in_provider_content_parts` and Java `rawBytesPreservedInProviderContentParts` assert image/audio/document request fields and video rejection. | | 5. Payload-safe output | New Python repr/string and Java string/sanitize assertions, plus existing serialization tests. | | 6. Base64 compatibility | Python factory preservation assertions and existing Python/Java Base64 payload-preservation tests. | Local results: Python API suite plus OpenAI/Ollama multimodal conversion tests: **449 passed, 14 skipped**. Focused Java API, cross-language snapshot and OpenAI conversion tests: **75 passed, 13 skipped**. Ruff, Spotless and whitespace checks passed. RAT passed with the Git-ignored local `.codex` directory excluded; the unmodified license script otherwise flags that local config file. Coverage targets binary corruption, accidental double encoding, mutable-array aliasing and wire-format drift. Not verified: live model requests, full repository suites, or live JVM/Python bridge E2E; cross-language coverage here uses serialized snapshots. <details> <summary>Validation commands and build evidence</summary> Python ran from `python/` with the existing `.venv`, `PYTHONPATH` set to that environment's `purelib`, and the source import path verified: ```sh .venv/bin/python -m pytest flink_agents/api/tests flink_agents/integrations/chat_models/openai/tests/test_openai_multimodal.py flink_agents/integrations/chat_models/tests/test_ollama_multimodal.py -q ``` Targeted Java reactor build/test recompiles the changed API and provider modules together: ```sh mvn -o -B --no-transfer-progress -pl api,integrations/chat-models/openai -am test -Dtest=MediaBlockBytesTest,ChatMessageSerializationTest,ChatMessageTest,CrossLanguageEventSnapshotTest,OpenAIChatCompletionsMultimodalTest -Dsurefire.failIfNoSpecifiedTests=false ``` </details> ### API Adds `from_bytes` / `fromBytes` on Image, Audio, Video and Document blocks. Existing Base64 and URL APIs, optional metadata conventions and serialized source types are unchanged. No new YAML syntax or binary wire representation is introduced. ### Documentation - [ ] `doc-needed` - [ ] `doc-not-needed` - [x] `doc-included` ### Was this patch authored or co-authored using generative AI tooling? - [x] Yes - [ ] No Generated-by: Codex CLI 0.153.4 (GPT-6) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
