mattcasters opened a new pull request, #7539: URL: https://github.com/apache/hop/pull/7539
## Summary Adds a **native Apache Spark 4.1.x pipeline engine** for Hop batch pipelines (**without** Apache Beam), so the community can try Dataset-based execution alongside the existing Beam Spark runner. ### Highlights - **Engine & run configuration** for Spark 4.1 (`local[*]` and cluster / `spark-submit` via `MainSpark`) - **Native handlers**: Memory Group By, Merge Join, Unique Rows, Sort Rows, Spark File Input/Output - **Generic mapPartitions** for other transforms (Filter, Switch/Case, Stream Lookup, Select Values, …) - **Info streams** (broadcast side data) for Stream Lookup and similar - **Target streams** for Filter Rows (true/false) and Switch/Case - **Multi-previous merge** when all input row layouts match (Beam Flatten equivalent) - **Execution information location** + sampling (driver metrics + executor samples) - **User manual**: [Getting started with the native Spark engine](docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/getting-started-with-native-spark.adoc) - **Integration tests**: `integration-tests/spark-native/` (0001–0006, 0099 complex) ### Notes for reviewers / try-out - Engine plugin: `plugins/engines/spark` - Prefer **Spark File Input/Output** (not classic Text File I/O) on this engine - `local[*]` uses Spark libs under `plugins/engines/spark/lib` - Cluster: use fat jar with `--spark-client-version=native-provided` and `org.apache.hop.spark.run.MainSpark` - Streaming / windows remain on **Beam** engines - Hop **Distribute vs Copy** is not implemented (same approach as Beam: target streams route; no round-robin) ## Test plan - [ ] `./mvnw -pl plugins/engines/spark test` - [ ] Run `integration-tests/spark-native` workflows (`main-0001` … `main-0006`, complex pipeline) - [ ] Create a **Native Spark** run configuration (`local[*]`) and run a small pipeline from Hop GUI - [ ] Optional: `spark-submit` with native-provided fat jar against a Spark 4.1 cluster - [ ] Spotless / RAT / docs build as applicable **Draft PR** for early community feedback and try-out before full review/merge. Related: #7486 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
