wenjin272 commented on code in PR #1252: URL: https://github.com/apache/flink-agents/pull/1252#discussion_r4271119254
########## docs/content/docs/development/calling-llms/structured-output.md: ########## @@ -0,0 +1,333 @@ +--- +title: "Structured Output" +weight: 8 +type: docs +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +## Overview + +Structured output makes a chat model return an object that matches a schema you define, instead of free text. A [`ChatResponseEvent`]({{< ref "docs/development/calling-llms/chat#handle-success-and-failure" >}}) carries the parsed object in `getStructuredOutput()` (Java) or `structured_output` (Python). + +There are two ways to get the model to follow the schema: + +- **Native:** the schema is sent in the provider's own structured-output parameter (for example OpenAI `response_format` or Anthropic `output_config`). The provider constrains the model's output to the schema. +- **Prompt:** the model is asked in the prompt text to reply in JSON that matches the schema, and the framework parses the reply. This works with every model, but nothing enforces the shape. + +You choose a *strategy*, and each provider connection reports whether it can apply the schema natively. The framework combines the two for every request. + +## Declare a Schema + +A schema is one of: + +| Schema | Python | Java | +|---|---|---| +| Typed class | Pydantic `BaseModel` subclass | POJO `Class` | +| Flink row type | `RowTypeInfo` | `RowTypeInfo` | + +Only a typed class can be applied natively. A `RowTypeInfo` always uses the prompt path. + +Pass the schema to a [ReAct Agent]({{< ref "docs/development/react_agent" >}}) with `output_schema`, or put it on a `ChatRequestEvent` (in Python, wrapped in `OutputSchema`) when you [send chat requests yourself]({{< ref "docs/development/calling-llms/chat" >}}). Review Comment: Could we clarify that Java also needs `new OutputSchema(rowTypeInfo)` when passing a `RowTypeInfo` to `ChatRequestEvent`? Only `ReActAgent` wraps it automatically. Passing the row type directly leaves it unchanged, and `ChatModelAction` fails with `Unsupported output schema`. Java POJO classes can still be passed directly. ########## docs/content/docs/development/calling-llms/structured-output.md: ########## @@ -0,0 +1,333 @@ +--- +title: "Structured Output" +weight: 8 +type: docs +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +## Overview + +Structured output makes a chat model return an object that matches a schema you define, instead of free text. A [`ChatResponseEvent`]({{< ref "docs/development/calling-llms/chat#handle-success-and-failure" >}}) carries the parsed object in `getStructuredOutput()` (Java) or `structured_output` (Python). + +There are two ways to get the model to follow the schema: + +- **Native:** the schema is sent in the provider's own structured-output parameter (for example OpenAI `response_format` or Anthropic `output_config`). The provider constrains the model's output to the schema. +- **Prompt:** the model is asked in the prompt text to reply in JSON that matches the schema, and the framework parses the reply. This works with every model, but nothing enforces the shape. + +You choose a *strategy*, and each provider connection reports whether it can apply the schema natively. The framework combines the two for every request. + +## Declare a Schema + +A schema is one of: + +| Schema | Python | Java | +|---|---|---| +| Typed class | Pydantic `BaseModel` subclass | POJO `Class` | +| Flink row type | `RowTypeInfo` | `RowTypeInfo` | + +Only a typed class can be applied natively. A `RowTypeInfo` always uses the prompt path. + +Pass the schema to a [ReAct Agent]({{< ref "docs/development/react_agent" >}}) with `output_schema`, or put it on a `ChatRequestEvent` (in Python, wrapped in `OutputSchema`) when you [send chat requests yourself]({{< ref "docs/development/calling-llms/chat" >}}). + +{{< tabs "Declare a Schema" >}} + +{{< tab "Python" >}} +```python +class Review(BaseModel): + summary: str + sentiment: Sentiment # an Enum, see "Enum Fields" below + +# ReAct Agent +agent = ReActAgent(chat_model=chat_model_descriptor, output_schema=Review) + +# Custom action +ctx.send_event( + ChatRequestEvent( + model="my_chat_model", + messages=[ + ChatMessage.user( + "Review the text below. Reply in JSON with the fields" + f" summary and sentiment.\n\n{text}" + ) + ], + output_schema=OutputSchema(output_schema=Review), + ) +) +``` +{{< /tab >}} + +{{< tab "Java" >}} +```java +public class Review { + public String summary; + public Sentiment sentiment; // an enum, see "Enum Fields" below +} + +// ReAct Agent +ReActAgent agent = new ReActAgent(chatModelDescriptor, prompt, Review.class); + +// Custom action +ctx.sendEvent( + new ChatRequestEvent( + "myChatModel", + List.of( + ChatMessage.user( + "Review the text below. Reply in JSON with the fields" + + " summary and sentiment.\n\n" + + text)), + Review.class)); +``` +{{< /tab >}} + +{{< /tabs >}} + +## Read the Result + +A ReAct Agent emits the parsed object as its output. In a custom action, read it from the successful `ChatResponseEvent`. For a typed class, Java returns an instance of the POJO. Python returns the model instance, or its fields as a `dict` once the event has been serialized, so validate it back into the model. For a `RowTypeInfo`, the result is a `Row`. + +{{< tabs "Read the Result" >}} + +{{< tab "Python" >}} +```python +@action(ChatResponseEvent.EVENT_TYPE) +@staticmethod +def process_response(event: Event, ctx: RunnerContext) -> None: + response = cast("ChatResponseEvent", event) + if response.is_failed: + raise RuntimeError(response.error) + review = Review.model_validate(response.structured_output) + ctx.send_event(OutputEvent(output=review.summary)) +``` +{{< /tab >}} + +{{< tab "Java" >}} +```java +@Action(EventType.ChatResponseEvent) +public static void processResponse(Event event, RunnerContext ctx) throws Exception { + ChatResponseEvent response = ChatResponseEvent.fromEvent(event); + if (response.isFailed()) { + throw new IllegalStateException(response.getError()); + } + Review review = (Review) response.getStructuredOutput(); + ctx.sendEvent(new OutputEvent(review.summary)); +} +``` +{{< /tab >}} + +{{< /tabs >}} + +## Choose a Strategy + +Set `structured_output_strategy` on the chat model **setup** (not the connection). The value is case-insensitive. + +| Strategy | Behavior | +|---|---| +| `AUTO` (default) | Native only when the connection recommends it for the model. Otherwise the prompt path, silently. | +| `NATIVE` | Native whenever the request can carry the schema, even for a model that is not on the connection's recommended list. Fails before any model call if it cannot. | +| `PROMPT` | Never native. | + +{{< tabs "Choose a Strategy" >}} + +{{< tab "Python" >}} +```python +ResourceDescriptor( + clazz=ResourceName.ChatModel.OPENAI_COMPLETIONS_SETUP, + connection="openai_connection", + model="gpt-4o-mini", + structured_output_strategy="native", +) +``` +{{< /tab >}} + +{{< tab "Java" >}} +```java +ResourceDescriptor.Builder.newBuilder(ResourceName.ChatModel.OPENAI_COMPLETIONS_SETUP) + .addInitialArgument("connection", "openaiConnection") + .addInitialArgument("model", "gpt-4o-mini") + .addInitialArgument("structured_output_strategy", "native") + .build(); +``` +{{< /tab >}} + +{{< /tabs >}} + +Each connection answers one of three things for a request: Review Comment: Could we keep these sections more focused on user-facing behavior? - **“Each connection answers one of three things…”** largely repeats the strategy table and exposes the internal capability classification. Could we fold the relevant fallback/error behavior into the `AUTO` / `NATIVE` / `PROMPT` descriptions? - **“How a Native Request Runs”** could focus on final-output-only constraints, the extra model call and its latency/cost, prompt requirements for custom actions, and failure handling. Details such as `chat_model_action`, durable records, tracing spans, and conversion-message construction seem better suited to developer documentation. Also, “up to two model calls” can be misleading with tool loops or retries; “one additional conversion call after the tool loop” would be clearer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
