88fantasy opened a new issue, #4479: URL: https://github.com/apache/streampark/issues/4479
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/streampark/issues?q=is%3Aissue+label%3A%22bug%22) and found no exact match for the current `dev` codebase. Note: #1215 (StreamX/StreamPark 1.15 era, closed 2024-07-16 with no linked fix, only a "copy the jar into lib/ manually" workaround) reports the *same symptom* (`NoClassDefFoundError: org/apache/calcite/sql/validate/SqlConformance`) but against pre-rewrite Scala code — this issue is a **new regression** introduced by the 3.0 Scala→Java shims migration (#4461), not a continuation of #1215. ### Java Version Temurin 21.0.11 (console), also reproduced building with Microsoft OpenJDK 11.0.28 ### Scala Version 2.12.x ### StreamPark Version 3.0.0-SNAPSHOT (`dev` branch, commit `f89652b67`) ### Flink Version 2.2.1 (official binary distribution, standalone/remote cluster) ### Deploy mode remote ### What happened `POST /flink/sql/verify` (the "Verify" button in the Flink SQL application editor) always fails — for every Flink version, not just 2.2.x — with two chained bugs on the `dev` branch: **1) `NoClassDefFoundError: org/apache/calcite/sql/validate/SqlConformance`** `FlinkShimsProxy` builds a per-Flink-version `ChildFirstClassLoader` to isolate SQL validation (`getVerifySqlLibClassLoader`) from the console's own runtime classpath. `FlinkSqlValidator` (which directly imports `org.apache.calcite.sql.validate.SqlConformance`) lives in `streampark-flink-shims-base`, a version-agnostic module that the console assembly also bundles directly onto its own `lib/` (and therefore its boot classpath). `FlinkShimsProxy#matchShimIncludeReason()` only matches jar names starting with `"streampark-flink-shims_flink"` (the per-version jars), so `streampark-flink-shims-base-*.jar` is never added to the per-version `ChildFirstClassLoader`'s own URLs. When the proxy loads `FlinkSqlValidator` through that classloader, `ChildFirstClassLoader.findClass()` fails to find it locally and falls back to the parent (the console's own `AppClassLoader`), which *does* have the jar on its boot classpath — so `FlinkSqlValidator` ends up defined by the console's `AppClassLoader` instead of the intended per-Flink-version classloader. The console's own classpath has no Calcite/table-planner jar, so any reflective inspection of `FlinkSqlValidator`'s methods (`Class.getDeclaredMethod`) throws `NoClassDefFoundError: org/apache/calcite/sql/validate/SqlConformance`. **2) `NotSerializableException: org.apache.streampark.flink.core.FlinkSqlValidationResult`** Once (1) is fixed and the shims classloader actually resolves `FlinkSqlValidator`, verification reaches `FlinkShimsProxy#getObject()`, which moves the result across the classloader boundary via plain Java serialization (`ObjectOutputStream` / `ClassLoaderObjectInputStream`). `FlinkSqlValidationResult` never implemented `Serializable`, so every *successful* validation still fails, this time with `NotSerializableException`. Both bugs were introduced by #4461 ("[Migrate] Migrate streampark-flink-shims from Scala to Java"), which is where `FlinkSqlValidator`/`FlinkSqlValidationResult` were rewritten into Java and moved into the newly split-out `streampark-flink-shims-base` module. ### Error Exception ``` java.lang.NoClassDefFoundError: org/apache/calcite/sql/validate/SqlConformance at org.apache.streampark.console.core.service.impl.FlinkSqlServiceImpl.lambda$verifySql$5(FlinkSqlServiceImpl.java:190) at org.apache.streampark.flink.proxy.FlinkShimsProxy.lambda$proxyVerifySql$1(FlinkShimsProxy.java:109) at org.apache.streampark.flink.proxy.FlinkShimsProxy.proxyVerifySql(FlinkShimsProxy.java:109) at org.apache.streampark.console.core.service.impl.FlinkSqlServiceImpl.verifySql(FlinkSqlServiceImpl.java:185) at org.apache.streampark.console.core.controller.FlinkSqlController.verify(FlinkSqlController.java:78) ... Caused by: java.lang.ClassNotFoundException: org.apache.calcite.sql.validate.SqlConformance at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526) ... 117 more ``` After patching (1) alone: ``` java.io.NotSerializableException: org.apache.streampark.flink.core.FlinkSqlValidationResult at java.base/java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1198) at java.base/java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:405) at org.apache.streampark.flink.proxy.FlinkShimsProxy.getObject(FlinkShimsProxy.java:117) ... ``` ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
