mattcasters commented on PR #7539:
URL: https://github.com/apache/hop/pull/7539#issuecomment-4994983587
## Changelog: Open table formats (lakehouse) series on native Spark
This commit adds optional **Delta Lake** / **Apache Iceberg** support on the
**native** Spark pipeline engine (not Beam). Summary of the planned PR stack
landed together as `cb9f5a2dcf`.
### Pins
| Component | Version |
|-----------|---------|
| Spark (engine) | 4.1.2 / Scala 2.13 |
| Delta Lake | `io.delta:delta-spark_4.1_2.13:4.3.1` |
| Apache Iceberg |
`org.apache.iceberg:iceberg-spark-runtime-4.1_2.13:1.11.0` |
**Distribution:** connectors are **not** in the default Hop assembly
(docs-only + operator install). Install under
`plugins/engines/spark/lib/{delta,iceberg}` + **restart Hop**, or use
`spark-submit --packages`. Sibling plugin classloaders are not enough (engine
TCCL).
---
### PR 1 — Packaging spike + connector probe
- Maven profile **`-Plakehouse`**: optional test deps +
`generate-test-resources` → frozen jar list under `target/lakehouse-connectors/`
- `SparkLakeConnectorProbe` with actionable missing-connector errors
- Spike findings documented in `plugins/engines/spark/README.md`:
- Delta PATH requires **extension + DeltaCatalog** (not extension alone)
- Iceberg bare `format("iceberg").load` routes toward HiveCatalog → use
**`hop_iceberg`** Hadoop catalog for PATH
### PR 2 — Lake Table Input/Output PATH (Delta)
- Transforms: **Spark Lake Table Input** / **Spark Lake Table Output** (Big
Data; native Spark only)
- Default Output save mode: **ErrorIfExists** (ACID-safe)
- Handlers registered in `HopPipelineMetaToSparkConverter`; action-sink /
empty-leaf write contract
- Unit tests without connector where possible; Delta PATH IT under
`-Plakehouse`
### PR 3 — Iceberg PATH Input/Output
- Symmetric PATH support for Iceberg via `writeTo(hop_iceberg.\`uri\`)` +
SQL read
- Built-in `hop_iceberg` catalog coexists with Delta on `spark_catalog`
- Iceberg PATH IT under `-Plakehouse`
### PR 4 — Time travel on Input
- Types: `NONE` / `VERSION` / `TIMESTAMP`
- Delta: `versionAsOf` / `timestampAsOf`; Iceberg: `VERSION AS OF` /
`TIMESTAMP AS OF`
- IT: write-twice / read-as-of
### PR 5 — Spark Catalog + LakeSessionPlan + TABLE mode
- Metadata type **Spark Catalog** (Connections): Hadoop / REST / custom /
advanced hive+glue
- `LakeSessionPlan` + `SparkCatalogApplier` wire conf into hop-run session
build (and active session where possible)
- TABLE mode on Input/Output (Iceberg-first; Delta TABLE advanced)
- Iceberg TABLE write locked to **`writeTo`**
### PR 6 — MERGE
- **Spark Lake Table Merge**: single upstream = source; target PATH or TABLE
- Structured actions: matched UPDATE_ALL/DELETE/NONE, not-matched
INSERT_ALL/NONE, optional not-matched-by-source DELETE
- Advanced raw MERGE SQL override; log-only metrics
- Delta PATH MERGE upsert IT
### PR 7 — Maintenance
- **Spark Lake Table Maintenance** (zero-input sink)
- Operations: OPTIMIZE / VACUUM / EXPIRE_SNAPSHOTS / REWRITE_MANIFESTS /
DELETE_WHERE
- Retention required for vacuum/expire; **acknowledge destructive** for
destructive ops
- Delta OPTIMIZE smoke IT
### PR 8 — Docs + samples
- User guide: `docs/.../pipeline/spark/lakehouse.adoc`
- Transform pages + `metadata-types/spark-catalog.adoc`; nav / indexes /
getting-started cross-links
- Operator samples: `integration-tests/spark-native/lakehouse/README.md`
- Engine README updated with full lakehouse section
---
### How to try
```bash
# Default tests (no connector download)
./mvnw -pl plugins/engines/spark -am test
# Lakehouse classpath + PATH/MERGE/maintenance ITs
./mvnw -pl plugins/engines/spark -am test -Plakehouse
```
Manual overlay: copy `target/lakehouse-connectors/*` into
`$HOP_HOME/plugins/engines/spark/lib/{delta,iceberg}` and restart Hop. Cluster:
`native-provided` fat jar + `--packages` (see lakehouse.adoc).
### Out of scope (optional later / PR 9)
- ASF-shipped connector overlay zip / fat-jar lakehouse tokens
- MDI, Hudi
- Default assembly embedding of Delta/Iceberg jars
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
