bamaer opened a new issue, #9020: URL: https://github.com/apache/hop/issues/9020
### Apache Hop version? 2.20.0-SNAPSHOT ### Java version? 21 ### Operating system Linux ### What happened? On the native Spark engine only one branch of a pipeline runs. Output transforms on the other branches (Text File Output, Table Output, ...) write nothing, and the run ends without an error. **Reproduce** Spark File Input → Filter Rows with a true and a false target → a Text File Output on each target, on `local[*]`. Only the false branch writes a file; the log ends with `Spark pipeline finished, result row count=<rows of that branch>`. The same happens with Switch / Case. **Cause** After building the Dataset graph the engine runs one `count()`, on the last leaf transform. Generic transforms are lazy `mapPartitions`, so the other leaves never run. Spark File Output and the lake table outputs are not affected: they write while the graph is built. Every branch that writes also recomputes the transforms before it. **Fix** Run an action on every leaf transform, and persist the output of transforms that several transforms read (Filter Rows / Switch / Case targets), so they run once. ### Issue Priority Priority: 2 ### Issue Component Component: Pipelines -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
