xintongsong commented on code in PR #446:
URL: https://github.com/apache/flink-agents/pull/446#discussion_r2700544389
##########
plan/src/main/java/org/apache/flink/agents/plan/actions/ChatModelAction.java:
##########
@@ -198,6 +199,8 @@ public static void chat(
(BaseChatModelSetup) ctx.getResource(model,
ResourceType.CHAT_MODEL);
boolean chatAsync =
ctx.getConfig().get(AgentExecutionOptions.CHAT_ASYNC);
+ // TODO: python chat model doesn't support async execution yet.
+ chatAsync = chatAsync && !(chatModel instanceof PythonChatModelSetup);
Review Comment:
It would be better to point to an issue that describes and tracks the
problem blocking python chat model from supporting async execution.
##########
plan/src/main/java/org/apache/flink/agents/plan/actions/ChatModelAction.java:
##########
@@ -210,7 +213,28 @@ public static void chat(
for (int attempt = 0; attempt < numRetries + 1; attempt++) {
try {
- response = chatModel.chat(messages, Map.of());
+ if (chatAsync) {
+ response =
+ ctx.durableExecuteAsync(
+ new DurableCallable<>() {
+ @Override
+ public String getId() {
+ return "chat-async";
+ }
+
+ @Override
+ public Class<ChatMessage>
getResultClass() {
+ return ChatMessage.class;
+ }
+
+ @Override
+ public ChatMessage call() throws
Exception {
+ return chatModel.chat(messages,
Map.of());
+ }
+ });
+ } else {
+ response = chatModel.chat(messages, Map.of());
Review Comment:
Shall we also use `durableExecute` for the synchronized model calls? Same
for the other resources and in python.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
