88fantasy opened a new issue, #4499: URL: https://github.com/apache/streampark/issues/4499
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/streampark/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues referencing Flink 2.x submission or `FlinkShimsProxy` jar matching. ### Java Version Temurin 21.0.11 (console), built with Microsoft OpenJDK 11.0.28 ### Scala Version 2.12.x ### StreamPark Version 3.0.0-SNAPSHOT (`dev` branch, commit `9ddda84c9`) ### Flink Version 2.2.1 (official binary distribution, standalone/remote cluster). Flink 1.20.4 works. ### Deploy mode remote ### What happened No Flink **2.x** SQL job can be submitted. Flink 1.x is unaffected. The failure moves as each layer is fixed, which is what made it look like an architectural limit rather than a set of bugs; it is five independent causes stacked on one path. **1 — `FlinkShimsProxy` stopped loading the version-specific shims jar.** `matchShimIncludeReason` matches on a jar name shaped `streampark-flink-shims_flink-<major>_<scala>`: ```java String prefixVer = FLINK_SHIMS_PREFIX + "-" + majorVersion + "_" + scalaVersion; return jarName.startsWith(prefixVer) ? "Include flink shims jar lib: " : null; ``` The artifacts carried that `_${scala.binary.version}` suffix until `e770d2e8e` renamed them without it. The jar is now `streampark-flink-shims_flink-2.2-3.0.0-SNAPSHOT.jar`, which does not match, so **the per-version shims jar is silently left out of the shims classloader**. There is no error — the classloader is simply built without it. **2 — The same rename silenced the rule that loads the rest of StreamPark's Flink jars.** ```java if (jarName.matches("^streampark-.*_" + scalaVersion + ".*$")) { ... } ``` Before the rename this selected `streampark-common`, `streampark-flink-client-api/core`, `streampark-flink-packer`, ... — everything the submission path is made of. Afterwards it selects only the Spark modules, which still carry the suffix. So `FlinkClientTrait`, `RemoteClient` and the `PackagedProgram` they use are loaded by the **console's own** classloader and resolve `org.apache.flink.*` against the console's fixed baseline Flink (1.20.1) instead of the registered target version. This is also, I believe, the real mechanism behind #4483: the `ServiceConfigurationError: ... not a subtype` reported there is what a half-populated shims classloader looks like from the outside — the interface comes from the baseline while `ServiceLoader` resolves the implementation through the target-version context classloader. **3 — `shims-base` and `shims-base-v2` are both loaded, in filesystem order.** They share **12 class names** — v2 redeclares them for Flink 2.x and inherits the rest — but `matchShimIncludeReason` matches both on the `streampark-flink-shims-base` prefix for *every* target version, and `addShimsUrls` adds them in `listFiles()` order. Which Flink version a class was compiled for is therefore decided by the filesystem: ``` org/apache/streampark/flink/core/conf/FlinkConfiguration.class org/apache/streampark/flink/core/FlinkSqlExecutor.class org/apache/streampark/flink/core/FlinkTableTrait.class ... (12 total) ``` **4 — Flink 2.x removed API this module compiles against.** Of the 53 Flink classes `streampark-flink-client-{api,core}` imports, only one is genuinely gone in 2.2.1: `org.apache.flink.runtime.jobgraph.SavepointConfigOptions` (2.x declares the same keys in `org.apache.flink.configuration.StateRecoveryOptions`). Along with it, `Configuration`'s typed accessors over a `ConfigOption` — `getBoolean` / `setBoolean` / `getInteger` — are gone; only the generic `get`/`set` remain. **5 — `ClusterClient#submitJob` changed signature.** `submitJob(JobGraph)` in 1.x, `submitJob(ExecutionPlan)` in 2.x. `JobGraph` implements `ExecutionPlan`, so the instance is fine — only the declared parameter type moved, which is enough for `NoSuchMethodError`. ### Error Exception ```log # with (1) and (2) unfixed — the class exists in the target's flink-dist, but the console's # baseline is consulted instead: Caused by: java.lang.NoClassDefFoundError: org/apache/flink/util/ParameterTool at org.apache.streampark.flink.core.TableContext.<init>(TableContext.java:51) at org.apache.streampark.flink.cli.SqlClient$BatchSqlApp.run(SqlClient.java:130) # after (1) and (2): java.lang.LinkageError: loader constraint violation: when resolving field "SAVEPOINT_PATH" ... org.apache.flink.runtime.jobgraph.SavepointConfigOptions is in unnamed module of loader 'app' # after (3) and (4): java.lang.NoSuchMethodError: 'java.util.concurrent.CompletableFuture org.apache.flink.client.program.ClusterClient.submitJob(org.apache.flink.runtime.jobgraph.JobGraph)' ``` ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
