weiqingy commented on issue #280: URL: https://github.com/apache/flink-agents/issues/280#issuecomment-5100189302
While working through the Azure step of this issue I looked at whether the `azureai` module (Azure AI Inference) can use native structured output. It cannot, at the currently pinned `azure-ai-inference:1.0.0-beta.5`, and I want to record why so nobody spends the same time on it again. There are two service-side constraints that contradict each other. The `json_schema` response format is rejected below api-version `2024-08-01-preview`. The service returns `400 "response_format value as json_schema is enabled only for api versions 2024-08-01-preview and later"`. But raising the SDK's service version to `V2024_08_01_PREVIEW` makes the endpoint return `404 "Resource not found"`. That 404 is why the SDK pinned its own default down: the beta.4 changelog entry is "Use ModelServiceVersion (api-version) of 2024-05-01-preview as default". `ModelServiceVersion` has exactly two values and `getLatest()` returns the older one. So one version routes but rejects the feature, and the other accepts the feature but does not route. There is no configuration of beta.5 that satisfies both. Worth flagging for anyone who tries: the `2024-05-01-preview` swagger does model `ChatCompletionsResponseFormatJsonSchema`, so the code compiles cleanly. Compiling is not evidence that the service accepts it. Azure closed the corresponding SDK issue as won't-fix, pointing at the beta SDK's retirement on 2026-08-26 and directing users to the OpenAI v1 API surface instead. I am therefore scoping the Azure work here to the Azure OpenAI connections, which use the OpenAI SDK on both the Java and Python sides and support the feature normally. Happy to revisit `azureai` if that module ever moves to the successor SDK. The Azure OpenAI half is up as #930. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
