mattcasters commented on PR #7539:
URL: https://github.com/apache/hop/pull/7539#issuecomment-4995075014
## Changelog: PR 9 — ship Delta/Iceberg in default assembly
Follow-up to the lakehouse series (PR 1–8): **connectors are now packaged
with the native Spark engine by default**.
### What changed
- **Runtime Maven deps** on `hop-engines-spark`:
- `io.delta:delta-spark_4.1_2.13:4.3.1` (+ kernel/storage/unitycatalog
tree)
- `org.apache.iceberg:iceberg-spark-runtime-4.1_2.13:1.11.0`
- Exclusions: Spark, Scala, Jackson core stack, slf4j (already on Hop /
spark lib)
- **Plugin zip layout**
- `plugins/engines/spark/lib/delta/*`
- `plugins/engines/spark/lib/iceberg/*`
- Staged via `dependency:copy-dependencies` →
`target/lakehouse-connectors` → assembly `fileSets` (full freeze list)
- **`-Plakehouse`** retained as a **no-op alias** for older scripts;
lakehouse ITs run on the default test classpath
- **Fat jar (`native-provided`)**: lakehouse jars are **not** excluded
(tests added in `HopBeamGuiPluginNativeFatJarTest`) — cluster submit no longer
needs `--packages` for connectors when using a current fat jar
- **Probe / docs / samples / NOTICE** updated for shipped packaging (manual
overlay is fallback only)
### How to verify
```bash
./mvnw -pl plugins/engines/spark -am package -DskipTests
unzip -l plugins/engines/spark/target/hop-engines-spark-*.zip | grep
'lib/delta\|lib/iceberg'
./mvnw -pl plugins/engines/spark test # 103 tests, includes lakehouse ITs
```
### Still optional / out of scope
- MDI for Lake Table transforms
- Hudi
- Separate ASF overlay zip (not needed now that jars ship in the engine
plugin)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
