wenjin272 commented on code in PR #919:
URL: https://github.com/apache/flink-agents/pull/919#discussion_r3671976274
##########
integrations/chat-models/openai/src/main/java/org/apache/flink/agents/integrations/chatmodels/openai/OpenAICompletionsConnection.java:
##########
@@ -119,11 +122,53 @@ public OpenAICompletionsConnection(
this.client = builder.build();
}
+ // Models for which OpenAI documents json_schema strict Structured Outputs
support.
+ // Source of truth:
https://platform.openai.com/docs/guides/structured-outputs — json_schema is
+ // supported on the gpt-4o-mini and gpt-4o-2024-08-06 snapshots "and
later"; gpt-4-turbo,
+ // earlier models, and gpt-3.5-turbo get JSON mode only.
+ //
+ // "and later" is temporal, not a name prefix: gpt-4o-2024-05-13 predates
the cutoff and does
+ // NOT support Structured Outputs, so matching a bare "gpt-4o" prefix
would misclassify it as
+ // capable and fail silently at the provider. Prefix matching is therefore
used only for the
+ // gpt-4o-mini family, whose entire lifetime post-dates the cutoff; every
other capable model is
+ // matched exactly. An unrecognized model reports not-capable and degrades
to the prompt
+ // fallback rather than failing at the provider.
+ private static final String NATIVE_STRUCTURED_OUTPUT_FAMILY_PREFIX =
"gpt-4o-mini";
+ private static final Set<String> NATIVE_STRUCTURED_OUTPUT_MODELS =
+ Set.of("gpt-4o", "gpt-4o-2024-08-06", "gpt-4o-2024-11-20");
+
+ @Override
+ protected boolean supportsNativeStructuredOutput(String effectiveModel) {
+ if (effectiveModel == null) {
+ return false;
+ }
+ return
effectiveModel.startsWith(NATIVE_STRUCTURED_OUTPUT_FAMILY_PREFIX)
+ || NATIVE_STRUCTURED_OUTPUT_MODELS.contains(effectiveModel);
Review Comment:
The capability predicate appears to be inaccurate in both directions.
Using `startsWith("gpt-4o-mini")` also classifies models such as
`gpt-4o-mini-audio-preview` and `gpt-4o-mini-realtime-preview` as capable,
although these variants do not support Structured Outputs. Conversely, models
such as `gpt-4.1`, `gpt-5`, `o3`, and `o4-mini` do support Structured Outputs
but currently return `false`.
Once #912 uses this predicate for `AUTO`, the false positives may produce
provider errors, while the false negatives will unnecessarily fall back to
prompting. Could we use stricter alias/snapshot matching, include the currently
supported model families, and add tests covering both cases?
References:
- https://developers.openai.com/api/docs/models/gpt-4o-mini-audio-preview
- https://developers.openai.com/api/docs/models/gpt-4.1
- https://developers.openai.com/api/docs/models/gpt-5
##########
integrations/chat-models/openai/src/main/java/org/apache/flink/agents/integrations/chatmodels/openai/OpenAICompletionsConnection.java:
##########
@@ -170,6 +220,13 @@ private ChatCompletionCreateParams buildRequest(
builder.tools(convertTools(tools, strictMode));
}
+ // Native structured output applies only for a POJO Class schema on a
model the provider
+ // documents as capable; a RowTypeInfo (wrapped in OutputSchema) or an
incapable model keeps
+ // the prompt-engineering fallback.
+ if (outputSchema instanceof Class &&
supportsNativeStructuredOutput(modelName)) {
Review Comment:
I agree that the final fix belongs in #912, where AUTO can be distinguished
from explicit NATIVE. Could we add a clear TODO referencing #912 at this
capability check, noting that NATIVE must bypass the connection-side fallback
or fail explicitly? This should help prevent the dormant issue from being
missed when the strategy wiring is implemented.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
