mattcasters commented on PR #7539:
URL: https://github.com/apache/hop/pull/7539#issuecomment-5002790392
## Follow-up: VFS vs Spark paths, project packages, and cluster mapping demo
Pushed additional work on this branch after the initial Native Spark engine
landings. Summary of improvements:
### Hop VFS vs Spark/Hadoop path dialects
- Documented two path languages clearly (Hop VFS `s3://` / named MinIO vs
Spark/Hadoop `s3a://`, `hdfs://`, …).
- Spark File / Lake path fields labeled as **Spark path (Hadoop URI)** with
tooltips.
- `SparkPathDialect` appends error hints when known Hop-only schemes appear
in Dataset I/O.
- Run-configuration **path scheme map** (`from=to`, e.g. `s3=s3a`) rewrites
schemes for Spark File / Lake PATH only (not classic VFS).
### Native Spark project package (nested pipelines / mappings)
- **Problem:** Simple Mapping / Pipeline Executor only store a path
(`${PROJECT_HOME}/…`); `MainSpark` shipped top-level `.hpl` + metadata only, so
children failed on cluster workers.
- **Package layout:** `project/` tree + root `metadata.json` + marker (skips
bulk `datasets/` by default).
- **Export:**
- CLI: `hop-conf --export-spark-project=… --export-spark-project-home=…`
(no project registration required; note `-j` enables a project,
`-p`/`--project` only names one for manage ops).
- GUI: **Tools → Export project package for Native Spark…**
- **Runtime:** `MainSpark --HopProjectPackage=… --HopPipelinePath=…`; engine
sets `PROJECT_HOME` after materialize under `java.io.tmpdir`.
- **Multi-host:** resolve package from SparkFiles →
**`$HOP_DATA/packages/<zip>`** (shared volume) → portable URI. Never stash
driver-local `…/userFiles-…` paths in variables. Demo also uses `spark-submit
--files` as backup.
- **Data vs definitions:** Dataset paths use `HOP_DATA` (or `s3a://…`);
package `PROJECT_HOME` is definitions only. Warnings when Spark File paths
still reference `PROJECT_HOME`.
### Docker walk-through (end-to-end proof)
- Sample: `plugins/engines/spark/src/samples/spark-mapping-demo` (parent +
Simple Mapping child + tiny CSV).
- Compose:
`docker/integration-tests/integration-tests-spark-native-cluster.yaml` — Spark
**4.1.2** master + workers, shared data volume.
- Scripts: `prepare-dist.sh` (default dist
**`/tmp/spark-mapping-demo-dist`**), `spark-submit-demo.sh` (`docker exec` on
master).
- Docs: *Cluster mapping walk-through* + paths-and-filesystems updates.
- Fix: Spark 4 master CLI uses `--host` (not Spark 3 `--ip`).
### Verified
Local multi-node Docker submit of `01-enrich-with-mapping.hpl` with nested
mapping under package `PROJECT_HOME` completed successfully.
Happy path (from repo root):
```bash
./plugins/engines/spark/src/samples/spark-mapping-demo/scripts/prepare-dist.sh
docker compose -f
docker/integration-tests/integration-tests-spark-native-cluster.yaml \
up -d --build --scale spark-worker=2
docker compose -f
docker/integration-tests/integration-tests-spark-native-cluster.yaml \
exec spark /opt/hop-samples/spark-mapping-demo/scripts/spark-submit-demo.sh
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
