weiqingy commented on code in PR #1040:
URL: https://github.com/apache/flink-agents/pull/1040#discussion_r3890854928
##########
plan/src/main/java/org/apache/flink/agents/plan/actions/ChatModelAction.java:
##########
@@ -576,8 +579,40 @@ private static Map<String, Object> takeRoutingMetadata(
return routing;
}
+ /**
+ * Rejects a response the provider did not finish emitting. A finish
reason reporting the
+ * content as cut off by the token budget or withheld by content filtering
raises {@link
+ * IllegalStateException}; any other reason, and an absent one, are
accepted.
+ */
+ private static void rejectIncompleteResponse(ChatMessage response) {
+ Object finishReason = response.getExtraArgs().get(FINISH_REASON);
+ if (TRUNCATED_FINISH_REASON.equals(finishReason)) {
+ throw new IllegalStateException(
+ String.format(
+ "ChatModel response is truncated
(finish_reason='%s'): it"
+ + " exhausted the completion token budget
before the model"
+ + " finished, so the content is incomplete
and cannot yield"
+ + " structured output. Raise the model's
max output tokens,"
+ + " or ask for a smaller output.",
+ finishReason));
+ }
+ if (CONTENT_FILTERED_FINISH_REASON.equals(finishReason)) {
+ throw new IllegalStateException(
+ String.format(
+ "ChatModel response was withheld by the provider's
content"
+ + " filter (finish_reason='%s'), so the
content is incomplete"
+ + " and cannot yield structured output.
Adjust the prompt or"
+ + " the provider's content filtering
configuration.",
+ finishReason));
+ }
+ }
+
static ChatMessage generateStructuredOutputWithReport(
RunnerContext ctx, ChatMessage response, Object outputSchema)
throws Exception {
+ // Precedes the start report: when this rejects, parsing is never
attempted, so there is no
+ // parser execution to report.
+ rejectIncompleteResponse(response);
Review Comment:
Yes, done in both languages.
Something I did not expect while tracing it: truncated tool call arguments
mostly never reach the gate. They fail earlier in the connection's converter
and come back as `MODEL_CALL_FAILED`. The one case the gate does catch is Java
only, where `parseArguments` turns blank arguments into `Map.of()` and the tool
runs with no arguments at all, while Python raises. That divergence looks
pre-existing. Worth its own issue, or would you rather it rode along here?
The trade is that truncated content no longer reaches the caller at all,
under any strategy, and a deterministic `length` now burns retries on every
call and again per router candidate. That still seems better than returning a
truncated answer that looks successful, but I would be glad to hear if you read
the retry cost differently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
