mattcasters commented on PR #7539:
URL: https://github.com/apache/hop/pull/7539#issuecomment-5004775155
### Update: Spark run mode, nested pipelines, and spark-demo expansion
Pushed follow-up work that has been exercised on the Docker Spark cluster.
**Engine**
- **Generic transform run mode** on Native Spark run config: `DISTRIBUTED`
(default) vs `DRIVER_ONLY`
- Per-transform override via context action **Spark Run Mode** (Inherit /
Force distributed / Force Driver Only), stored on `TransformMeta.attributes`
(`spark` / `run_mode`)
- Bottom-left canvas badge for forced modes
(`DrawSparkRunModeOnTransformExtensionPoint`)
- Nested **Pipeline Executor → Native Spark child**: reuses parent
`SparkSession`; nested engines do not stop/cancel the shared session
- Warning when Pipeline Executor runs DISTRIBUTED with nested Spark intent
**Execution info**
- `CacheEntry.writeToDisk` now uses **Hop VFS** so roots like
`file:///data/hop-data/executions` (from `${HOP_DATA}/executions`) work
- Unit coverage for `file://` root folder writes
**spark-demo** (renamed from `spark-mapping-demo`)
- `01` mapping package demo (existing)
- `02` Workflow Executor → country folders + instance markers
- `03` nested Native Spark pipelines per country (`Force Driver Only`)
- Execution info location + data profile sample metadata; host GUI vars in
`spark-demo-gui-config.json`
- Compose: host bind mount **`/tmp/spark-demo-dist/hop-data` →
`/data/hop-data`** so `out/` and `executions/` are visible without `docker exec`
**Docs**
- Native Spark run config + cluster walk-through updates, including Force
Driver Only screenshot
Happy path on the host after a run:
```bash
ls -la /tmp/spark-demo-dist/hop-data/out
ls -la /tmp/spark-demo-dist/hop-data/executions
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
