88fantasy opened a new issue, #4483: URL: https://github.com/apache/streampark/issues/4483
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/streampark/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues referencing `PipelineExecutorFactory` or `ServiceConfigurationError`. ### Java Version Temurin 21.0.11 ### Scala Version 2.12.x ### StreamPark Version 3.0.0-SNAPSHOT (`dev` branch, commit `f89652b67`) ### Flink Version 2.2.1 (official binary distribution, standalone/remote cluster) ### Deploy mode remote ### What happened **This is a report of a likely root cause, not a fix** — I'm filing it separately from #4479 / #4480 / #4481 / #4482 because it appears to require touching `FlinkShimsProxy`, which this repo's own `AGENTS.md` marks as a "High-Sensitivity Area" ("Never introduce static state that could leak across classloader boundaries"), and the right fix has real architectural tradeoffs I don't think I should pick unilaterally. After #4482's fix unblocks `FlinkClient#submit()` past its `SecurityManager` setup, job submission for a Flink SQL application against a real Flink 2.2.1 standalone cluster fails one step later with: ``` java.util.ServiceConfigurationError: org.apache.flink.core.execution.PipelineExecutorFactory: org.apache.flink.client.deployment.executors.RemoteExecutorFactory not a subtype ``` thrown from `org.apache.flink.client.cli.GenericCLI.getExecutorFactoryNames`, called from `FlinkClientTrait.getCustomCommandLines` → `FlinkClientTrait.submit`. **Suspected mechanism**: the console's own boot classpath bundles a concrete Flink baseline (`flink-clients-1.20.1.jar`, `flink-core-1.20.1.jar`, etc. — visible in the running process's `-classpath`), separate from the per-Flink-version jars that `FlinkShimsProxy`'s `ChildFirstClassLoader` loads for the target cluster (here, Flink 2.2.1). Both `flink-clients-1.20.1.jar` and the target cluster's `flink-clients-2.2.1.jar` declare the same `META-INF/services/org.apache.flink.core.execution.PipelineExecutorFactory` service, naming the same fully-qualified class `org.apache.flink.client.deployment.executors.RemoteExecutorFactory` in both. `ChildFirstClassLoader#getResources()` (in `streampark-common`) collects results from **both** its own `findResources()` **and** `parent.getResources()` for every resource lookup, including `META-INF/services/*` files that `ServiceLoader` reads. That means `ServiceLoader.load(PipelineExecutorFactory.class)` running under the per-Flink-2.2.1-version classloader can end up resolving a service entry whose declaring jar and whose *actually loaded* `RemoteExecutorFactory`/`PipelineExecutorFactory` `Class` objects come from two different classloaders (the console's own `AppClassLoader`, via the 1.20.1 baseline, vs. the per-version `ChildFirstClassLoader`, via the 2.2.1 shims) — despite having identical fully-qualified names, two `Class` objects loaded by different classloaders are different types to the JVM, hence `... not a subtype`. I have **not** verified a fix, and see at least two different directions with different tradeoffs, which is why I think this needs maintainer input rather than a PR from me: 1. Stop bundling a concrete Flink version's client jars on the console's own boot classpath (if nothing there actually needs them at that level), so `ServiceLoader` lookups under the per-version classloader only ever see one Flink version's service declarations. 2. Change `ChildFirstClassLoader#getResources()` to not merge parent results for `org.apache.flink.*` resource lookups specifically, so `ServiceLoader` running under a per-version classloader is fully isolated from whatever Flink version(s) happen to be on the console's own classpath. ### Error Exception ``` java.util.concurrent.CompletionException: java.util.ServiceConfigurationError: org.apache.flink.core.execution.PipelineExecutorFactory: org.apache.flink.client.deployment.executors.RemoteExecutorFactory not a subtype at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315) ... Caused by: java.util.ServiceConfigurationError: org.apache.flink.core.execution.PipelineExecutorFactory: org.apache.flink.client.deployment.executors.RemoteExecutorFactory not a subtype at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:593) at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1244) ... at org.apache.flink.client.cli.GenericCLI.getExecutorFactoryNames(GenericCLI.java:136) at org.apache.flink.client.cli.GenericCLI.<init>(GenericCLI.java:59) at org.apache.flink.client.cli.CliFrontend.loadCustomCommandLines(CliFrontend.java:1425) at org.apache.streampark.flink.client.trait.FlinkClientTrait.getCustomCommandLines(FlinkClientTrait.java:590) at org.apache.streampark.flink.client.trait.FlinkClientTrait.getCommandLineOptions(FlinkClientTrait.java:604) at org.apache.streampark.flink.client.trait.FlinkClientTrait.getCommandLineAndFlinkConfig(FlinkClientTrait.java:704) at org.apache.streampark.flink.client.trait.FlinkClientTrait.prepareConfig(FlinkClientTrait.java:325) at org.apache.streampark.flink.client.trait.FlinkClientTrait.submit(FlinkClientTrait.java:302) at org.apache.streampark.flink.client.FlinkClientEntrypoint.submit(FlinkClientEntrypoint.java:84) at org.apache.streampark.flink.client.FlinkClient.invokeClient(FlinkClient.java:110) at org.apache.streampark.flink.client.FlinkClient.submit(FlinkClient.java:64) ... ``` ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! *(diagnosis only for now — see explanation above; happy to attempt a fix once the direction is agreed)* ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
