88fantasy opened a new issue, #4481: URL: https://github.com/apache/streampark/issues/4481
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/streampark/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues referencing `BuildPipeline`/`execStep`. ### Java Version Temurin 21.0.11 (console), also reproduced building with Microsoft OpenJDK 11.0.28 ### Scala Version 2.12.x ### StreamPark Version 3.0.0-SNAPSHOT (`dev` branch, commit `f89652b67`) ### Flink Version 2.2.1 (official binary distribution, standalone/remote cluster) ### Deploy mode remote ### What happened Building/releasing a Flink SQL application always fails at step 1/2 ("Create building workspace") with an uncaught `NullPointerException`, even though the workspace directory is genuinely created successfully on disk. The failure is **not visible anywhere in the normal log files** (`logs/error.*.log`) — it only shows up in the raw stdout redirect (`logs/streampark.out`), because it's thrown from a background thread pool task before the console's `GlobalExceptionHandler` is ever involved, and the persisted `ApplicationBuildPipeline.hasError`/`errorSummary`/`errorStack` fields stay `null`/`false`, making the failure look silent from the REST API (`POST /flink/pipe/detail` just shows `pipeStatus: FAILURE` with no error detail). Root cause: `BuildPipeline#execStep()` wraps every successful step result in `Optional.of(result)`: ```java R result = process.call(); ... return java.util.Optional.of(result); // <-- throws NPE if result == null ``` `Optional.of(null)` always throws `NullPointerException` per the Java API contract (use `Optional.ofNullable` for that). Many pipeline steps legitimately have nothing meaningful to return — e.g. "create/clean the build workspace" just performs a side effect and returns `null` — and this is the *normal*, successful case, not an edge case. This is a classic Scala→Java migration bug (introduced by #4463, "[Migrate] Migrate streampark-flink-packer from Scala to Java"): the original Scala code almost certainly used `Option(x)`, which tolerates `null` and produces `None`, not the strict `Optional.of(x)`. This is not specific to Flink SQL or remote/standalone deploy mode — the same `execStep(...).orElseThrow(() -> pipelineException())` pattern, with a step whose `Callable` returns `null`, appears at **~20 call sites across 6 files**: `BuildPipeline#runYarnSqlBuildSteps` (used by the Yarn pipelines), `FlinkRemoteBuildPipeline`, `FlinkK8sSessionBuildPipeline`, `AbstractK8sApplicationBuildPipeline`, `FlinkK8sApplicationBuildPipeline`, and `SparkK8sApplicationBuildPipeline`. Every deploy mode's build pipeline is affected wherever its first (or any null-returning) step runs. Simply swapping `Optional.of` for `Optional.ofNullable` is **not sufficient** on its own: every call site relies on `.orElseThrow(() -> pipelineException())` to turn a *failed* step into a thrown exception, and `Optional<R>` cannot distinguish "empty because the step failed" from "empty because the step succeeded with a `null` result" — so a naive one-line fix just trades a crash for `.orElseThrow()` incorrectly firing on every successful null-returning step. The actual fix needs `execStep()` to signal failure via a thrown exception directly (matching what all 20 call sites already reconstruct via `orElseThrow`) rather than via `Optional` emptiness, and the corresponding call sites need to drop the now-redundant `.orElseThrow(...)`. ### Error Exception ``` 06:31:23.581 [streampark-build-pipeline-executor-0] ERROR org.apache.streampark.flink.packer.pipeline.impl.FlinkRemoteBuildPipeline - [StreamPark] [streampark-packer] Building pipeline has failed. | appName=streampark_bounded_probe java.lang.IllegalStateException: null at org.apache.streampark.flink.packer.pipeline.BuildPipeline.pipelineException(BuildPipeline.java:259) at org.apache.streampark.flink.packer.pipeline.impl.FlinkRemoteBuildPipeline.lambda$buildProcess$1(FlinkRemoteBuildPipeline.java:64) at java.base/java.util.Optional.orElseThrow(Optional.java:403) at org.apache.streampark.flink.packer.pipeline.impl.FlinkRemoteBuildPipeline.buildProcess(FlinkRemoteBuildPipeline.java:63) ... Caused by: java.lang.NullPointerException: null at java.base/java.util.Objects.requireNonNull(Objects.java:233) at java.base/java.util.Optional.of(Optional.java:113) at org.apache.streampark.flink.packer.pipeline.BuildPipeline.execStep(BuildPipeline.java:135) at org.apache.streampark.flink.packer.pipeline.impl.FlinkRemoteBuildPipeline.buildProcess(FlinkRemoteBuildPipeline.java:59) ... 5 common frames omitted ``` ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
