sangkyoonnam opened a new issue, #1292: URL: https://github.com/apache/flink-agents/issues/1292
### Search before asking - [x] I searched in the [issues](https://github.com/apache/flink-agents/issues) and found nothing similar. ### Description The released `flink-agents-dist` JARs I checked are built with JDK 17 and don't contain the Flink Agents JDK 21 continuation overlay. On main, awaiting Java `ctx.chat` (#1220) and resolving an internal sub-agent call from Java (#1138) require that runtime, so a distribution built with JDK 17 rejects them even on JDK 21 with `--add-exports=java.base/jdk.internal.vm=ALL-UNNAMED`. A 0.4 release built the same way would reject these calls. I found this during the #1283 walkthrough. The overlay is compiled by the `java-21` profile in `runtime/pom.xml`, which activates only when the build JDK is 21 or later (#438), while `tools/releasing/deploy_staging_jars.sh` requires JDK 17 (#413). `create_binary_release.sh` doesn't pin a JDK, but the released binaries also report JDK 17. All the released JARs I inspected have `Build-Jdk-Spec: 17` and no Flink Agents classes under `META-INF/versions/21/`: `flink-agents-dist-flink-2.1-0.2.1.jar` and `flink-agents-dist-flink-2.2-0.3.0.jar` from Maven Central, `flink-agents-dist-flink-2.3-0.3.1.jar` from dlcdn, and the JARs in the 0.3.1 wheel. CI builds and tests on JDK 21, and `ChatCallIntegrationTest`, enabled on JDK 21+, asserts that the executor loads from `!/META-INF/versions/21/` in a packaged JAR. In `release-0.3.1`, the base executor's `executeAsync` runs the supplier synchronously. On main, the affected paths require the continuation runtime: without it, `ChatDurableFuture.await` throws `UnsupportedOperationException`, and `InternalSubagentSetup` throws `IllegalStateException`. I ran a minimal Java job that awaits `ctx.chat` through a fake connection on a Flink 2.3.0 session cluster on JDK 21.0.11, with the `--add-exports` option in `env.java.opts.all`: - JDK 17-built main distribution (`a50a8a18`), no overlay: the job fails with `java.lang.UnsupportedOperationException: Java chat calls require the JDK 21 continuation runtime`. - JDK 21-built main distribution (`742faf60`), with the overlay: the job finishes, the TaskManager stdout has `chat-ok:echo:alpha` and `chat-ok:echo:beta`, and the log has `Initialize fixed thread pool for async task`. The continuation runtime, the chat guard and the build configuration are the same in both commits. I verified the sub-agent guard from the code, not with a separate run. For 0.4, the release artifacts need the overlay, either by building the release on JDK 21 or by compiling `src/main/java21` with a JDK 21 toolchain while Maven runs on JDK 17 (the current profile won't activate on its own then). In my JDK 21 build the base API, plan and runtime classes stay Java 11 bytecode and only the overlay is Java 21, but compatibility on older runtimes should still be checked. The wheel needs a look too: the thin JARs' shade filter includes only `org/apache/flink/agents/**`, so even in my JDK 21 build the thin JAR has the base executor and no overlay, while the common JAR has the overlay and no base class. I haven't tested which one a PyFlink job loads. A release check that the distribution JARs contain both overlay classes and that JDK 21 loads the versioned executor would catch this before a release. I can send a PR for the build change and the artifact check. For 0.4, would you prefer building the release on JDK 21, or compiling the overlay with a JDK 21 toolchain while Maven runs on JDK 17? cc @Sxnan @yunfengzhou-hub ### How to reproduce 1. `unzip -l flink-agents-dist-flink-2.3-0.3.1.jar | grep versions/21/org/apache/flink/agents` prints nothing, and its `META-INF/MANIFEST.MF` has `Build-Jdk-Spec: 17`. 2. Build main in two clean worktrees, one with JDK 17 and one with JDK 21 (`mvn clean install -DskipTests`), and keep each `dist/flink-2.3/target/flink-agents-dist-flink-2.3-0.4-SNAPSHOT.jar` (not the `-thin` one). Only the JDK 21 JAR has `META-INF/versions/21/org/apache/flink/agents/runtime/async/ContinuationActionExecutor.class`. 3. Save the job below as `ChatJob.java` and package it: ```sh mkdir -p classes javac -cp "<dist JAR>:$FLINK_HOME/lib/*" -d classes ChatJob.java jar --create --file chatjob.jar -C classes . ``` For each dist JAR, stop the cluster, put that JAR in `$FLINK_HOME/lib` in place of the other, append `--add-exports=java.base/jdk.internal.vm=ALL-UNNAMED` to `env.java.opts.all`, start Flink 2.3.0 on JDK 21, and run `$FLINK_HOME/bin/flink run -c demo.ChatJob chatjob.jar`. 4. With the JDK 17-built JAR the job fails as above. With the JDK 21-built JAR it finishes and the results are in the TaskManager stdout. <details> <summary>Minimal job</summary> ```java package demo; import org.apache.flink.agents.api.AgentsExecutionEnvironment; import org.apache.flink.agents.api.Event; import org.apache.flink.agents.api.EventType; import org.apache.flink.agents.api.InputEvent; import org.apache.flink.agents.api.OutputEvent; import org.apache.flink.agents.api.agents.Agent; import org.apache.flink.agents.api.annotation.Action; import org.apache.flink.agents.api.annotation.ChatModelConnection; import org.apache.flink.agents.api.annotation.ChatModelSetup; import org.apache.flink.agents.api.chat.messages.ChatMessage; import org.apache.flink.agents.api.chat.messages.ChatResult; import org.apache.flink.agents.api.chat.model.BaseChatModelConnection; import org.apache.flink.agents.api.chat.model.BaseChatModelSetup; import org.apache.flink.agents.api.context.RunnerContext; import org.apache.flink.agents.api.resource.ResourceContext; import org.apache.flink.agents.api.resource.ResourceDescriptor; import org.apache.flink.api.java.functions.KeySelector; import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; import java.util.HashMap; import java.util.List; import java.util.Map; /** Minimal Java ctx.chat job with a fake chat connection. */ public class ChatJob { public static void main(String[] args) throws Exception { StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); env.setParallelism(1); AgentsExecutionEnvironment agents = AgentsExecutionEnvironment.getExecutionEnvironment(env); agents.fromDataStream(env.fromElements("alpha", "beta"), new KeyByValue()) .apply(new ChatAgent()) .toDataStream() .print(); agents.execute(); } public static class KeyByValue implements KeySelector<String, String> { @Override public String getKey(String value) { return value; } } public static class ChatAgent extends Agent { @ChatModelConnection public static ResourceDescriptor connection() { return ResourceDescriptor.Builder.newBuilder(EchoConnection.class.getName()).build(); } @ChatModelSetup public static ResourceDescriptor model() { return ResourceDescriptor.Builder.newBuilder(EchoModel.class.getName()) .addInitialArgument("connection", "connection") .addInitialArgument("model", "echo") .build(); } @Action(EventType.InputEvent) public static void call(Event event, RunnerContext ctx) throws Exception { String key = String.valueOf(InputEvent.fromEvent(event).getInput()); ChatResult result = ctx.chat("model", List.of(ChatMessage.user(key))).await(); ctx.sendEvent(new OutputEvent("chat-ok:" + result.getText())); } } public static class EchoModel extends BaseChatModelSetup { public EchoModel(ResourceDescriptor descriptor, ResourceContext context) { super(descriptor, context); } @Override public Map<String, Object> getParameters() { return new HashMap<>(); } } public static class EchoConnection extends BaseChatModelConnection { public EchoConnection(ResourceDescriptor descriptor, ResourceContext context) { super(descriptor, context); } @Override public ChatResult chat( List<ChatMessage> messages, List<org.apache.flink.agents.api.tools.Tool> tools, Map<String, Object> params) { return new ChatResult(ChatMessage.assistant("echo:" + messages.get(0).getText())); } } } ``` </details> ### Version and environment JDK 17-built main at `a50a8a18` and JDK 21-built main at `742faf60`; the relevant runtime and build configuration are unchanged at `bff15845`. Flink 2.3.0 standalone session cluster, run on Corretto 21.0.11, built with Corretto 17.0.14 and 21.0.11, macOS 15.3.1 on Apple M4. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
