weiqingy commented on code in PR #1252: URL: https://github.com/apache/flink-agents/pull/1252#discussion_r4277282152
########## docs/content/docs/development/calling-llms/structured-output.md: ########## @@ -0,0 +1,333 @@ +--- +title: "Structured Output" +weight: 8 +type: docs +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +## Overview + +Structured output makes a chat model return an object that matches a schema you define, instead of free text. A [`ChatResponseEvent`]({{< ref "docs/development/calling-llms/chat#handle-success-and-failure" >}}) carries the parsed object in `getStructuredOutput()` (Java) or `structured_output` (Python). + +There are two ways to get the model to follow the schema: + +- **Native:** the schema is sent in the provider's own structured-output parameter (for example OpenAI `response_format` or Anthropic `output_config`). The provider constrains the model's output to the schema. +- **Prompt:** the model is asked in the prompt text to reply in JSON that matches the schema, and the framework parses the reply. This works with every model, but nothing enforces the shape. + +You choose a *strategy*, and each provider connection reports whether it can apply the schema natively. The framework combines the two for every request. + +## Declare a Schema + +A schema is one of: + +| Schema | Python | Java | +|---|---|---| +| Typed class | Pydantic `BaseModel` subclass | POJO `Class` | +| Flink row type | `RowTypeInfo` | `RowTypeInfo` | + +Only a typed class can be applied natively. A `RowTypeInfo` always uses the prompt path. + +Pass the schema to a [ReAct Agent]({{< ref "docs/development/react_agent" >}}) with `output_schema`, or put it on a `ChatRequestEvent` (in Python, wrapped in `OutputSchema`) when you [send chat requests yourself]({{< ref "docs/development/calling-llms/chat" >}}). + +{{< tabs "Declare a Schema" >}} + +{{< tab "Python" >}} +```python +class Review(BaseModel): + summary: str + sentiment: Sentiment # an Enum, see "Enum Fields" below + +# ReAct Agent +agent = ReActAgent(chat_model=chat_model_descriptor, output_schema=Review) + +# Custom action +ctx.send_event( + ChatRequestEvent( + model="my_chat_model", + messages=[ + ChatMessage.user( + "Review the text below. Reply in JSON with the fields" + f" summary and sentiment.\n\n{text}" + ) + ], + output_schema=OutputSchema(output_schema=Review), + ) +) +``` +{{< /tab >}} + +{{< tab "Java" >}} +```java +public class Review { + public String summary; + public Sentiment sentiment; // an enum, see "Enum Fields" below +} + +// ReAct Agent +ReActAgent agent = new ReActAgent(chatModelDescriptor, prompt, Review.class); + +// Custom action +ctx.sendEvent( + new ChatRequestEvent( + "myChatModel", + List.of( + ChatMessage.user( + "Review the text below. Reply in JSON with the fields" + + " summary and sentiment.\n\n" + + text)), + Review.class)); +``` +{{< /tab >}} + +{{< /tabs >}} + +## Read the Result + +A ReAct Agent emits the parsed object as its output. In a custom action, read it from the successful `ChatResponseEvent`. For a typed class, Java returns an instance of the POJO. Python returns the model instance, or its fields as a `dict` once the event has been serialized, so validate it back into the model. For a `RowTypeInfo`, the result is a `Row`. + +{{< tabs "Read the Result" >}} + +{{< tab "Python" >}} +```python +@action(ChatResponseEvent.EVENT_TYPE) +@staticmethod +def process_response(event: Event, ctx: RunnerContext) -> None: + response = cast("ChatResponseEvent", event) + if response.is_failed: + raise RuntimeError(response.error) + review = Review.model_validate(response.structured_output) + ctx.send_event(OutputEvent(output=review.summary)) +``` +{{< /tab >}} + +{{< tab "Java" >}} +```java +@Action(EventType.ChatResponseEvent) +public static void processResponse(Event event, RunnerContext ctx) throws Exception { + ChatResponseEvent response = ChatResponseEvent.fromEvent(event); + if (response.isFailed()) { + throw new IllegalStateException(response.getError()); + } + Review review = (Review) response.getStructuredOutput(); + ctx.sendEvent(new OutputEvent(review.summary)); +} +``` +{{< /tab >}} + +{{< /tabs >}} + +## Choose a Strategy + +Set `structured_output_strategy` on the chat model **setup** (not the connection). The value is case-insensitive. + +| Strategy | Behavior | +|---|---| +| `AUTO` (default) | Native only when the connection recommends it for the model. Otherwise the prompt path, silently. | +| `NATIVE` | Native whenever the request can carry the schema, even for a model that is not on the connection's recommended list. Fails before any model call if it cannot. | +| `PROMPT` | Never native. | + +{{< tabs "Choose a Strategy" >}} + +{{< tab "Python" >}} +```python +ResourceDescriptor( + clazz=ResourceName.ChatModel.OPENAI_COMPLETIONS_SETUP, + connection="openai_connection", + model="gpt-4o-mini", + structured_output_strategy="native", +) +``` +{{< /tab >}} + +{{< tab "Java" >}} +```java +ResourceDescriptor.Builder.newBuilder(ResourceName.ChatModel.OPENAI_COMPLETIONS_SETUP) + .addInitialArgument("connection", "openaiConnection") + .addInitialArgument("model", "gpt-4o-mini") + .addInitialArgument("structured_output_strategy", "native") + .build(); +``` +{{< /tab >}} + +{{< /tabs >}} + +Each connection answers one of three things for a request: Review Comment: Thanks, agreed. Fixed in baad1ced: - Removed the "Each connection answers one of three things" list. The fallback and error behavior now sits in the `AUTO` / `NATIVE` / `PROMPT` rows, and the "AUTO falls back silently" hint stays. - Renamed the section to "How Native Output Behaves" and rewrote it around the final-output-only schema, the extra conversion call and its latency and cost, the prompt a custom action must write, and parse failures. `chat_model_action`, durable records, tracing spans and the conversion message are gone. - Replaced "up to two model calls" with "one conversion call after the tool loop". - Provider Support no longer uses the "feasible/infeasible" wording; it says "blocked". I kept a short note on how the ReAct Agent changes its prompt per path, since it affects what users see. WDYT? ########## docs/content/docs/development/calling-llms/structured-output.md: ########## @@ -0,0 +1,333 @@ +--- +title: "Structured Output" +weight: 8 +type: docs +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +## Overview + +Structured output makes a chat model return an object that matches a schema you define, instead of free text. A [`ChatResponseEvent`]({{< ref "docs/development/calling-llms/chat#handle-success-and-failure" >}}) carries the parsed object in `getStructuredOutput()` (Java) or `structured_output` (Python). + +There are two ways to get the model to follow the schema: + +- **Native:** the schema is sent in the provider's own structured-output parameter (for example OpenAI `response_format` or Anthropic `output_config`). The provider constrains the model's output to the schema. +- **Prompt:** the model is asked in the prompt text to reply in JSON that matches the schema, and the framework parses the reply. This works with every model, but nothing enforces the shape. + +You choose a *strategy*, and each provider connection reports whether it can apply the schema natively. The framework combines the two for every request. + +## Declare a Schema + +A schema is one of: + +| Schema | Python | Java | +|---|---|---| +| Typed class | Pydantic `BaseModel` subclass | POJO `Class` | +| Flink row type | `RowTypeInfo` | `RowTypeInfo` | + +Only a typed class can be applied natively. A `RowTypeInfo` always uses the prompt path. + +Pass the schema to a [ReAct Agent]({{< ref "docs/development/react_agent" >}}) with `output_schema`, or put it on a `ChatRequestEvent` (in Python, wrapped in `OutputSchema`) when you [send chat requests yourself]({{< ref "docs/development/calling-llms/chat" >}}). Review Comment: Good catch, thanks. I confirmed it in the code: only `ReActAgent` wraps a `RowTypeInfo`, and `ChatModelAction` throws `Unsupported output schema` for a bare one. Fixed in baad1ced. The guide now says that on a `ChatRequestEvent`, Python wraps the schema in `OutputSchema`, and Java passes a POJO class directly but must use `new OutputSchema(rowTypeInfo)` for a `RowTypeInfo`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
