This is an automated email from the ASF dual-hosted git repository.
hansva pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hop.git
The following commit(s) were added to refs/heads/main by this push:
new 475bc0c8c2 Issue #5524 : Document the lake table transforms on the Hop
engine (#8859)
475bc0c8c2 is described below
commit 475bc0c8c2cb3b30c6ff185c40eef528378aa2f8
Author: Ankur Kotwal <[email protected]>
AuthorDate: Tue Oct 13 03:59:35 2026 -0700
Issue #5524 : Document the lake table transforms on the Hop engine (#8859)
* Issue #5524 : Document the lake table transforms on the Hop engine
Lake table input and output now run on the Hop engine as well, so they
lose the "Spark" in their names and their documentation explains what
they do there: finding PATH and TABLE tables, deletes, types, copies,
one commit per pipeline run, the save modes and how PATH commits behave
on object stores. The catalog type becomes "Lakehouse catalog"; its key
stays spark-catalog. Merge and maintenance still run on Spark only and
keep their names.
The iceberg integration tests get a local engine round trip, the same
scenario as the existing Spark one.
Generated-by: Claude Opus 5.5
* Issue #5524 : Address review of the lake table docs
- Navigation, the transform and metadata indexes and all cross-links use
the new names (Lake table input/output, Lakehouse catalog); the list
entries are re-sorted under L.
- Dates and timestamps: the docs now say what the code does. Values
without a time zone are read and written as wall-clock time in the Hop
JVM's zone, as Parquet Input and Output do, not as UTC; timestamptz
values are instants.
- BigNumber columns: the field's length and precision are used only when
the length is greater than 0, otherwise decimal(38, 10).
- Concurrent PATH commits: every local file system is safe (hard link,
or exclusive create without hard links); only HDFS, object stores and
other non-local VFS locations need a single writer.
- The pages' marketplace id hop-tech-lakehouse resolves now that #8854
lists the plugin in the catalog; a catalog test pins that entry, and
both transform pages say which plugins they need.
- The 0004 validation pipeline describes the 0004 extract.
Generated-by: Claude Opus 5.5
* Issue #5524 : Say which lake table transforms run on the Hop engine in
the Spark lakehouse guide
Generated-by: Claude Opus 5.5
---
docs/hop-user-manual/modules/ROOT/nav.adoc | 6 +-
.../metadata-types/databricks-connection.adoc | 2 +-
.../modules/ROOT/pages/metadata-types/index.adoc | 2 +-
.../ROOT/pages/metadata-types/spark-catalog.adoc | 21 +--
.../native-spark-pipeline-engine.adoc | 2 +-
.../ROOT/pages/pipeline/spark/databricks.adoc | 2 +-
.../spark/getting-started-with-native-spark.adoc | 8 +-
.../ROOT/pages/pipeline/spark/lakehouse.adoc | 28 ++--
.../pipeline/spark/paths-and-filesystems.adoc | 4 +-
.../modules/ROOT/pages/pipeline/transforms.adoc | 4 +-
.../pipeline/transforms/spark-file-input.adoc | 2 +-
.../pipeline/transforms/spark-file-output.adoc | 2 +-
.../transforms/spark-lake-table-input.adoc | 51 ++++--
.../transforms/spark-lake-table-maintenance.adoc | 4 +-
.../transforms/spark-lake-table-merge.adoc | 6 +-
.../transforms/spark-lake-table-output.adoc | 82 ++++++++--
integration-tests/iceberg/0004-read-local.hpl | 154 ++++++++++++++++++
integration-tests/iceberg/0004-validation.hpl | 167 +++++++++++++++++++
integration-tests/iceberg/0004-write-local.hpl | 120 ++++++++++++++
integration-tests/iceberg/README.md | 5 +-
.../iceberg/main-0004-local-input-output.hwf | 180 +++++++++++++++++++++
.../metadata/unit-test/0004-validation UNIT.json | 32 ++++
.../catalog/OptionalPluginCatalogTest.java | 1 +
.../metadata/messages/messages_en_US.properties | 4 +-
.../transforms/messages/messages_en_US.properties | 28 ++--
25 files changed, 827 insertions(+), 90 deletions(-)
diff --git a/docs/hop-user-manual/modules/ROOT/nav.adoc
b/docs/hop-user-manual/modules/ROOT/nav.adoc
index 73922ce413..90b4630e79 100644
--- a/docs/hop-user-manual/modules/ROOT/nav.adoc
+++ b/docs/hop-user-manual/modules/ROOT/nav.adoc
@@ -232,6 +232,8 @@ under the License.
*** xref:pipeline/transforms/jsonoutput.adoc[JSON Output]
*** xref:pipeline/transforms/kafkaconsumer.adoc[Kafka consumer]
*** xref:pipeline/transforms/kafkaproducer.adoc[Kafka producer]
+*** xref:pipeline/transforms/spark-lake-table-input.adoc[Lake table input]
+*** xref:pipeline/transforms/spark-lake-table-output.adoc[Lake table output]
*** xref:pipeline/transforms/languagemodelchat.adoc[Language model chat]
*** xref:pipeline/transforms/ldapinput.adoc[LDAP Input]
*** xref:pipeline/transforms/ldapoutput.adoc[LDAP Output]
@@ -310,10 +312,8 @@ under the License.
*** xref:pipeline/transforms/sortedschemamerge.adoc[Sorted schema merge]
*** xref:pipeline/transforms/spark-file-input.adoc[Spark file input]
*** xref:pipeline/transforms/spark-file-output.adoc[Spark file output]
-*** xref:pipeline/transforms/spark-lake-table-input.adoc[Spark lake table
input]
*** xref:pipeline/transforms/spark-lake-table-maintenance.adoc[Spark lake
table maintenance]
*** xref:pipeline/transforms/spark-lake-table-merge.adoc[Spark lake table
merge]
-*** xref:pipeline/transforms/spark-lake-table-output.adoc[Spark lake table
output]
*** xref:pipeline/transforms/spark-sql.adoc[Spark SQL]
*** xref:pipeline/transforms/splitfields.adoc[Split Fields]
*** xref:pipeline/transforms/splitfieldtorows.adoc[Split fields to rows]
@@ -546,6 +546,7 @@ under the License.
** xref:metadata-types/graphs/graph-database-connection.adoc[Graph Database
Connection]
** xref:metadata-types/hdfs-connection.adoc[HDFS Connection]
** xref:metadata-types/hop-server.adoc[Hop Server]
+** xref:metadata-types/spark-catalog.adoc[Lakehouse catalog]
** xref:metadata-types/mail-server-connection.adoc[Mail Server Connection]
** xref:metadata-types/masking-pattern.adoc[Masking Pattern]
** xref:metadata-types/minio-connection.adoc[Minio (S3) Connection]
@@ -565,7 +566,6 @@ under the License.
** xref:metadata-types/smb-connection.adoc[SMB Connection]
** xref:metadata-types/splunk-connection.adoc[Splunk Connection]
** xref:metadata-types/jms-connection.adoc[JMS connection]
-** xref:metadata-types/spark-catalog.adoc[Spark Catalog]
** xref:metadata-types/variable-resolver/index.adoc[Variable Resolver]
***
xref:metadata-types/variable-resolver/aws-secrets-manager-variable-resolver.adoc[AWS
Secrets Manager variable resolver]
***
xref:metadata-types/variable-resolver/azure-key-vault-variable-resolver.adoc[Azure
Key Vault variable resolver]
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/metadata-types/databricks-connection.adoc
b/docs/hop-user-manual/modules/ROOT/pages/metadata-types/databricks-connection.adoc
index 52ede2a816..4bc2468445 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/metadata-types/databricks-connection.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/metadata-types/databricks-connection.adoc
@@ -76,4 +76,4 @@ See xref:vfs/databricks-vfs.adoc[Databricks Volumes /
Workspace VFS].
* xref:workflow/actions/databricks-job-run.adoc[Databricks job run] — trigger
and wait for Jobs API runs
* xref:database/databases/databricks.adoc[Databricks database (JDBC)] — SQL
warehouse access
* xref:pipeline/spark/getting-started-with-native-spark.adoc[Native Spark
pipeline engine] — fat jar / MainSpark on cluster
-* xref:metadata-types/spark-catalog.adoc[Spark Catalog] — Unity Catalog
templates for lakehouse TABLE mode
+* xref:metadata-types/spark-catalog.adoc[Lakehouse catalog] — Unity Catalog
templates for lakehouse TABLE mode
diff --git a/docs/hop-user-manual/modules/ROOT/pages/metadata-types/index.adoc
b/docs/hop-user-manual/modules/ROOT/pages/metadata-types/index.adoc
index 13995493e3..92ba47d136 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/metadata-types/index.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/metadata-types/index.adoc
@@ -71,7 +71,7 @@ xref:metadata-types/data-stream/data-stream.adoc[Data
Stream]: A Data Stream can
* xref:metadata-types/rest-connection.adoc[REST Connection]: Describes all the
metadata needed to connect to a REST api.
* xref:metadata-types/salesforce-connection.adoc[Salesforce Connection]:
Describes a reusable Salesforce connection that can be used across multiple
Salesforce transforms
* xref:metadata-types/splunk-connection.adoc[Splunk Connection]: Describes a
Splunk connection
-* xref:metadata-types/spark-catalog.adoc[Spark Catalog]: Spark SQL catalog
configuration for Delta Lake / Apache Iceberg TABLE mode on the native Spark
engine
+* xref:metadata-types/spark-catalog.adoc[Lakehouse catalog]: catalog of Apache
Iceberg and Delta Lake tables for the lake table transforms in TABLE mode, on
the Hop engine and the native Spark engine
* xref:metadata-types/static-schema-definition.adoc[Static Schema Definition]:
Defines a reusable data stream layout to ensure consistency across multiple
pipelines and simplify schema management.
* xref:metadata-types/variable-resolver/index.adoc[Variable Resolver]: Use
plugins to resolve variable values with a pipeline, a key store, a vaults, or
secret managers.
* xref:hop-server/web-service.adoc[Web Service]: Allows to run a pipeline to
generate output for a servlet on Hop Server
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/metadata-types/spark-catalog.adoc
b/docs/hop-user-manual/modules/ROOT/pages/metadata-types/spark-catalog.adoc
index 96ecd26dc2..86148b5511 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/metadata-types/spark-catalog.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/metadata-types/spark-catalog.adoc
@@ -16,18 +16,21 @@ under the License.
////
:imagesdir: ../../assets/images/
:page-pagination:
-:description: Spark Catalog metadata configures Spark SQL catalogs for Delta
Lake and Apache Iceberg TABLE mode on the native Spark pipeline engine.
-:page-marketplace: hop-engines-spark
+:description: Lakehouse catalog metadata describes the catalog of Apache
Iceberg and Delta Lake tables for the lake table transforms in TABLE mode, on
the Hop engine and on the native Spark pipeline engine.
+:page-marketplace: hop-tech-lakehouse
-= Spark Catalog
+= Lakehouse catalog
== Description
-*Spark Catalog* is a reusable Hop metadata type (category **Connections**)
that describes a Spark SQL catalog for open table formats on the
xref:pipeline/spark/getting-started-with-native-spark.adoc[native Spark
pipeline engine].
+*Lakehouse catalog* (called *Spark Catalog* in earlier versions) is a reusable
Hop metadata type (category **Connections**) that describes a catalog for open
table formats. Lake table transforms in **TABLE** mode reference a catalog by
name.
-Lake Table transforms in **TABLE** mode can reference a Spark Catalog by name.
Hop expands the entry into `spark.sql.catalog.<catalogName>.*` configuration
when building or preparing the session (`LakeSessionPlan` /
`SparkCatalogApplier`).
+* On the xref:pipeline/spark/getting-started-with-native-spark.adoc[native
Spark pipeline engine] Hop expands the entry into
`spark.sql.catalog.<catalogName>.*` configuration when building or preparing
the session (`LakeSessionPlan` / `SparkCatalogApplier`).
+* On the Hop engine, xref:pipeline/transforms/spark-lake-table-input.adoc[Lake
table input] and xref:pipeline/transforms/spark-lake-table-output.adoc[Lake
table output] open the catalog with the Iceberg Java library. The `hadoop` and
`rest` catalog types are supported there. The REST catalog gets the *URI*,
*Warehouse* and *Credential* (as `token`), plus the *Extra conf* lines as
catalog properties; Spark-style keys (`spark.sql.catalog.<name>.<key>`) are
accepted, so one catalog definiti [...]
-PATH-mode Iceberg tables do **not** require this metadata type — the engine
registers a Hadoop catalog for the folder that holds each table
(`hop_iceberg_<hash>`). Use Spark Catalog for production Iceberg warehouses,
REST catalogs, and other named catalogs.
+Existing catalogs keep working: the metadata key (`spark-catalog`) hasn't
changed.
+
+PATH-mode Iceberg tables do **not** require this metadata type — the engine
registers a Hadoop catalog for the folder that holds each table
(`hop_iceberg_<hash>`). Use a lakehouse catalog for production Iceberg
warehouses, REST catalogs, and other named catalogs.
Connector packaging and session matrix:
xref:pipeline/spark/lakehouse.adoc[Lakehouse tables on the native Spark engine].
@@ -141,11 +144,11 @@ When both Delta and Iceberg are used in one session, keep
**Delta** on `spark_ca
== spark-submit
-Export project metadata JSON that includes every Spark Catalog entry
referenced by the pipeline (and any used only via run configuration).
Alternatively, pass equivalent `--conf spark.sql.catalog.…` flags on the submit
command. Paths and warehouses must be reachable from the driver and executors.
+Export project metadata JSON that includes every lakehouse catalog entry
referenced by the pipeline (and any used only via run configuration).
Alternatively, pass equivalent `--conf spark.sql.catalog.…` flags on the submit
command. Paths and warehouses must be reachable from the driver and executors.
== Related pages
* xref:pipeline/spark/lakehouse.adoc[Lakehouse tables on the native Spark
engine]
-* xref:pipeline/transforms/spark-lake-table-input.adoc[Spark lake table input]
-* xref:pipeline/transforms/spark-lake-table-output.adoc[Spark lake table
output]
+* xref:pipeline/transforms/spark-lake-table-input.adoc[Lake table input]
+* xref:pipeline/transforms/spark-lake-table-output.adoc[Lake table output]
* xref:pipeline/spark/getting-started-with-native-spark.adoc[Getting started
with the native Spark pipeline engine]
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-run-configurations/native-spark-pipeline-engine.adoc
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-run-configurations/native-spark-pipeline-engine.adoc
index 8bb210e5b0..aaba21c560 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-run-configurations/native-spark-pipeline-engine.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-run-configurations/native-spark-pipeline-engine.adoc
@@ -57,7 +57,7 @@ Use **Load configuration template** to fill the fields below
from a common deplo
Templates replace the current engine options after confirmation when the form
already has custom values. The form fields update immediately after you apply a
template. Edit values afterward as needed.
-For named Iceberg/Delta catalogs used by Lake Table transforms in **TABLE**
mode, see xref:metadata-types/spark-catalog.adoc[Spark Catalog] (**Load catalog
template**).
+For named Iceberg/Delta catalogs used by Lake Table transforms in **TABLE**
mode, see xref:metadata-types/spark-catalog.adoc[Lakehouse catalog] (**Load
catalog template**).
[options="header",cols="1,3,1"]
|===
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/databricks.adoc
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/databricks.adoc
index 7b47216021..3b2f02ac7b 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/databricks.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/databricks.adoc
@@ -692,4 +692,4 @@ Databricks does **not** expose a standard content MD5/SHA
for volume files compa
*
xref:pipeline/pipeline-run-configurations/native-spark-pipeline-engine.adoc[Native
Spark run configuration]
* xref:pipeline/spark/paths-and-filesystems.adoc[Paths and file systems]
* xref:database/databases/databricks.adoc[Databricks JDBC] (SQL warehouse —
separate use case)
-* xref:metadata-types/spark-catalog.adoc[Spark Catalog] (Unity Catalog
templates for lakehouse TABLE mode)
+* xref:metadata-types/spark-catalog.adoc[Lakehouse catalog] (Unity Catalog
templates for lakehouse TABLE mode)
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/getting-started-with-native-spark.adoc
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/getting-started-with-native-spark.adoc
index 3d2a627d2e..fd9fc36309 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/getting-started-with-native-spark.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/getting-started-with-native-spark.adoc
@@ -157,7 +157,7 @@ This is the fastest way to learn: Spark runs *inside* the
Hop process using all
TIP: On the Native Spark engine options tab, **Load configuration template**
can pre-fill common setups (*Local execution*, *Spark standalone server*,
*spark-submit*, *Databricks*, *YARN*, …). See
xref:pipeline/pipeline-run-configurations/native-spark-pipeline-engine.adoc[Native
Spark pipeline engine].
-TIP: For Iceberg **TABLE** mode, create a
xref:metadata-types/spark-catalog.adoc[Spark Catalog] entry and use **Load
catalog template** (Hadoop local/object store, REST, Hive, Glue, Nessie, …).
Each preset starts **Extra conf** with a commented `# docs:` link to vendor
documentation.
+TIP: For Iceberg **TABLE** mode, create a
xref:metadata-types/spark-catalog.adoc[Lakehouse catalog] entry and use **Load
catalog template** (Hadoop local/object store, REST, Hive, Glue, Nessie, …).
Each preset starts **Extra conf** with a commented `# docs:` link to vendor
documentation.
. Open *Metadata* → *Pipeline run configuration* → *New*.
. Give it a clear name, for example `spark-local`.
@@ -445,9 +445,9 @@ Writes run as Spark *actions* when the engine materialises
the graph. Prefer *Sp
Classic *Spark File* transforms cover `csv` / `parquet` / `json` / `orc` /
`text` only. For ACID lake tables use the dedicated **Lake Table** transforms
(Big Data category):
-* xref:pipeline/transforms/spark-lake-table-input.adoc[Spark lake table input]
/ xref:pipeline/transforms/spark-lake-table-output.adoc[Spark lake table output]
+* xref:pipeline/transforms/spark-lake-table-input.adoc[Lake table input] /
xref:pipeline/transforms/spark-lake-table-output.adoc[Lake table output]
* xref:pipeline/transforms/spark-lake-table-merge.adoc[Spark lake table merge]
/ xref:pipeline/transforms/spark-lake-table-maintenance.adoc[Spark lake table
maintenance]
-* xref:metadata-types/spark-catalog.adoc[Spark Catalog] metadata for Iceberg
TABLE mode
+* xref:metadata-types/spark-catalog.adoc[Lakehouse catalog] metadata for
Iceberg TABLE mode
Connector JARs **ship in the default Hop assembly** under
`plugins/engines/spark/lib/{delta,iceberg}` (and ride a *native-provided* fat
jar for cluster submit). Full session conf, PATH vs TABLE, and samples:
xref:pipeline/spark/lakehouse.adoc[Lakehouse tables on the native Spark engine].
@@ -711,6 +711,6 @@ image::pipeline/spark/spark-native-0099-complex.png[0099
complex sample,width=95
*
xref:pipeline/pipeline-run-configurations/pipeline-run-configurations.adoc[Pipeline
run configurations]
* xref:metadata-types/execution-information-location.adoc[Execution
information location]
* xref:metadata-types/execution-data-profile.adoc[Execution data profile]
-* xref:metadata-types/spark-catalog.adoc[Spark Catalog]
+* xref:metadata-types/spark-catalog.adoc[Lakehouse catalog]
* xref:hop-gui/perspective-execution-information.adoc[Execution Information
perspective]
* xref:hop-tools/hop-conf/hop-conf.adoc[hop-conf] (fat jar, metadata export)
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/lakehouse.adoc
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/lakehouse.adoc
index e44db08e66..7f798e1254 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/lakehouse.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/lakehouse.adoc
@@ -33,11 +33,11 @@ It assumes you already have a working *Native Spark*
pipeline run configuration
|===
| Capability | How
-| Read / write ACID tables |
xref:pipeline/transforms/spark-lake-table-input.adoc[Spark lake table input] /
xref:pipeline/transforms/spark-lake-table-output.adoc[Spark lake table output]
+| Read / write ACID tables |
xref:pipeline/transforms/spark-lake-table-input.adoc[Lake table input] /
xref:pipeline/transforms/spark-lake-table-output.adoc[Lake table output]
| Path-based tables | Identifier mode **PATH** (local, HDFS, object-store URIs
Spark can open)
-| Catalog tables | Identifier mode **TABLE** +
xref:metadata-types/spark-catalog.adoc[Spark Catalog] metadata (Iceberg-first).
Use **Load catalog template** on the metadata editor for
Hadoop/REST/Hive/Glue/Nessie skeletons.
+| Catalog tables | Identifier mode **TABLE** +
xref:metadata-types/spark-catalog.adoc[Lakehouse catalog] metadata
(Iceberg-first). Use **Load catalog template** on the metadata editor for
Hadoop/REST/Hive/Glue/Nessie skeletons.
| Time travel | Version / snapshot id or timestamp on Input
@@ -126,7 +126,7 @@ export SPARK_HOME=/path/to/spark-4.1.x-bin-hadoop3
./hop-conf.sh --generate-fat-jar=/tmp/hop-native-provided.jar \
--spark-client-version=native-provided
-# Export project metadata (run config + any Spark Catalog entries) to JSON,
then:
+# Export project metadata (run config + any lakehouse catalog entries) to
JSON, then:
"${SPARK_HOME}/bin/spark-submit" \
--master spark://your-master:7077 \
--conf
spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension,org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
\
@@ -163,7 +163,7 @@ Hop’s `LakeSessionPlan` scans Lake Table transforms before
the session is used
|Iceberg extensions + a Hadoop catalog per table folder,
**`hop_iceberg_<hash>`** (`type=hadoop`, warehouse = the folder that holds the
table)
|Iceberg TABLE
-|`spark.sql.catalog.<name>=org.apache.iceberg.spark.SparkCatalog` (+ `type`,
`warehouse` / `uri`, …) from xref:metadata-types/spark-catalog.adoc[Spark
Catalog] metadata
+|`spark.sql.catalog.<name>=org.apache.iceberg.spark.SparkCatalog` (+ `type`,
`warehouse` / `uri`, …) from xref:metadata-types/spark-catalog.adoc[Lakehouse
catalog] metadata
|===
IMPORTANT: Delta PATH I/O on Spark 4.1 + Delta 4.3 **requires DeltaCatalog**,
not only the extension. Iceberg bare `format("iceberg").load(path)` is **not**
used — it defaults toward HiveCatalog and fails without HMS. PATH mode uses
`hop_iceberg_<hash>.\`<table>\``, where the catalog's warehouse is the table's
parent folder, so the table is read and written at its path.
@@ -184,7 +184,7 @@ NOTE: Before Hop 2.20, Iceberg PATH tables written by the
Spark engine were stor
|Local lakes, object-store folders, quick prototypes
|**TABLE**
-|Multi-part name (`catalog.namespace.table`) + optional Spark Catalog metadata
name
+|Multi-part name (`catalog.namespace.table`) + optional lakehouse catalog
metadata name
|Production catalogs (Iceberg Hadoop warehouse, REST, …)
|===
@@ -226,10 +226,10 @@ NOTE: Before Hop 2.20, Iceberg PATH tables written by the
Spark engine were stor
== Lake Table transforms
-All four live under the **Big Data** category and are **native Spark only**
(not Beam Spark, not the local Hop engine).
+All four live under the **Big Data** category. Lake table merge and Lake table
maintenance run on the native Spark engine only; Lake table input and Lake
table output also run on the Hop engine (Apache Iceberg only). None of them run
on Beam engines.
-* xref:pipeline/transforms/spark-lake-table-input.adoc[Spark lake table input]
— read, optional projection, time travel
-* xref:pipeline/transforms/spark-lake-table-output.adoc[Spark lake table
output] — append / overwrite / …; **default save mode is ErrorIfExists**
+* xref:pipeline/transforms/spark-lake-table-input.adoc[Lake table input] —
read, optional projection, time travel
+* xref:pipeline/transforms/spark-lake-table-output.adoc[Lake table output] —
append / overwrite / …; **default save mode is ErrorIfExists**
* xref:pipeline/transforms/spark-lake-table-merge.adoc[Spark lake table merge]
— single upstream hop is the source; target is PATH or TABLE
* xref:pipeline/transforms/spark-lake-table-maintenance.adoc[Spark lake table
maintenance] — zero-input action (OPTIMIZE, VACUUM, expire, …)
@@ -252,12 +252,12 @@ Injection keys are documented on each transform page.
Shared identity keys: `FOR
. Install Delta jars (see above) and restart Hop.
. Create a *Native Spark* run configuration (`local[*]`).
-. Pipeline: *Row Generator* (or *Spark file input*) → *Spark lake table output*
+. Pipeline: *Row Generator* (or *Spark file input*) → *Lake table output*
** Format: `delta`
** Identifier mode: `PATH`
** Table path: e.g. `+${PROJECT_HOME}/output/delta-orders+`
** Save mode: `Overwrite` for a first write (default is *ErrorIfExists*)
-. Second pipeline or hop: *Spark lake table input* on the same path → *Spark
file output* or preview.
+. Second pipeline or hop: *Lake table input* on the same path → *Spark file
output* or preview.
. Run with the native Spark run configuration.
=== Time travel sketch
@@ -295,7 +295,7 @@ Standalone *Spark lake table maintenance*:
|Do not use bare `format("iceberg")` outside Hop; ensure Iceberg transforms
run under hop-run so the per-folder `hop_iceberg_<hash>` catalogs are registered
|TABLE unresolved
-|Spark Catalog metadata exported? Catalog name matches the first segment of
`catalog.ns.table`? Warehouse / URI reachable from driver and executors?
+|Lakehouse catalog metadata exported? Catalog name matches the first segment
of `catalog.ns.table`? Warehouse / URI reachable from driver and executors?
|Destructive maintenance rejected
|Set retention hours / WHERE and enable acknowledge flag
@@ -308,9 +308,9 @@ Standalone *Spark lake table maintenance*:
* xref:pipeline/spark/getting-started-with-native-spark.adoc[Getting started
with the native Spark pipeline engine]
*
xref:pipeline/pipeline-run-configurations/native-spark-pipeline-engine.adoc[Native
Spark pipeline engine]
-* xref:metadata-types/spark-catalog.adoc[Spark Catalog]
-* xref:pipeline/transforms/spark-lake-table-input.adoc[Spark lake table input]
-* xref:pipeline/transforms/spark-lake-table-output.adoc[Spark lake table
output]
+* xref:metadata-types/spark-catalog.adoc[Lakehouse catalog]
+* xref:pipeline/transforms/spark-lake-table-input.adoc[Lake table input]
+* xref:pipeline/transforms/spark-lake-table-output.adoc[Lake table output]
* xref:pipeline/transforms/spark-lake-table-merge.adoc[Spark lake table merge]
* xref:pipeline/transforms/spark-lake-table-maintenance.adoc[Spark lake table
maintenance]
* Developer notes and spike findings: `plugins/engines/spark/README.md` in the
Hop source tree
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/paths-and-filesystems.adoc
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/paths-and-filesystems.adoc
index d543c09d72..e25db56193 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/paths-and-filesystems.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/paths-and-filesystems.adoc
@@ -65,7 +65,7 @@ For the full list of Hop VFS schemes, see
xref:vfs.adoc[Virtual File Systems].
|**Spark URI**
|e.g. `file:///…`, `s3a://bucket/table` — see
xref:pipeline/spark/lakehouse.adoc[Lakehouse]
-|Spark Catalog warehouse / Iceberg `io-impl`
+|Lakehouse catalog warehouse / Iceberg `io-impl`
|**Spark URI**
|Templates use `s3a://` and `file:///`
@@ -310,4 +310,4 @@ See
xref:pipeline/spark/getting-started-with-native-spark.adoc[Getting started w
*
xref:pipeline/pipeline-run-configurations/native-spark-pipeline-engine.adoc[Native
Spark pipeline engine]
* xref:vfs.adoc[Virtual File Systems]
* xref:metadata-types/minio-connection.adoc[Minio Connection]
-* xref:metadata-types/spark-catalog.adoc[Spark Catalog]
+* xref:metadata-types/spark-catalog.adoc[Lakehouse catalog]
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms.adoc
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms.adoc
index 4844594aad..f6c5433db3 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms.adoc
@@ -157,6 +157,8 @@ Those are explained once in
xref:pipeline/formatting-values.adoc[Formatting numb
* xref:pipeline/transforms/jsonoutput.adoc[JSON Output]
* xref:pipeline/transforms/kafkaconsumer.adoc[Kafka consumer]
* xref:pipeline/transforms/kafkaproducer.adoc[Kafka producer]
+* xref:pipeline/transforms/spark-lake-table-input.adoc[Lake table input]
+* xref:pipeline/transforms/spark-lake-table-output.adoc[Lake table output]
* xref:pipeline/transforms/languagemodelchat.adoc[Language model chat]
* xref:pipeline/transforms/ldapinput.adoc[LDAP Input]
* xref:pipeline/transforms/ldapoutput.adoc[LDAP Output]
@@ -229,10 +231,8 @@ Those are explained once in
xref:pipeline/formatting-values.adoc[Formatting numb
* xref:pipeline/transforms/sort.adoc[Sort Rows]
* xref:pipeline/transforms/sortedmerge.adoc[Sorted Merge]
* xref:pipeline/transforms/sortedschemamerge.adoc[Sorted schema merge]
-* xref:pipeline/transforms/spark-lake-table-input.adoc[Spark lake table input]
* xref:pipeline/transforms/spark-lake-table-maintenance.adoc[Spark lake table
maintenance]
* xref:pipeline/transforms/spark-lake-table-merge.adoc[Spark lake table merge]
-* xref:pipeline/transforms/spark-lake-table-output.adoc[Spark lake table
output]
* xref:pipeline/transforms/spark-sql.adoc[Spark SQL]
* xref:pipeline/transforms/splitfields.adoc[Split Fields]
* xref:pipeline/transforms/splitfieldtorows.adoc[Split fields to rows]
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/spark-file-input.adoc
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/spark-file-input.adoc
index 75e53e46c7..dd8e201f84 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/spark-file-input.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/spark-file-input.adoc
@@ -94,5 +94,5 @@ Field group `FIELDS`: `NAME`, `TYPE`, `LENGTH`, `PRECISION`,
`FORMAT_MASK`.
* xref:pipeline/transforms/spark-file-output.adoc[Spark file output]
* xref:pipeline/spark/paths-and-filesystems.adoc[Paths and file systems on the
native Spark engine]
-* xref:pipeline/transforms/spark-lake-table-input.adoc[Spark lake table input]
for Delta Lake and Apache Iceberg tables
+* xref:pipeline/transforms/spark-lake-table-input.adoc[Lake table input] for
Delta Lake and Apache Iceberg tables
*
xref:pipeline/pipeline-run-configurations/native-spark-pipeline-engine.adoc[Native
Spark pipeline engine]
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/spark-file-output.adoc
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/spark-file-output.adoc
index bde439f3d7..fa9b4f711e 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/spark-file-output.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/spark-file-output.adoc
@@ -88,5 +88,5 @@ All options support
xref:pipeline/metadata-injection.adoc[metadata injection].
* xref:pipeline/transforms/spark-file-input.adoc[Spark file input]
* xref:pipeline/spark/paths-and-filesystems.adoc[Paths and file systems on the
native Spark engine]
-* xref:pipeline/transforms/spark-lake-table-output.adoc[Spark lake table
output] for Delta Lake and Apache Iceberg tables
+* xref:pipeline/transforms/spark-lake-table-output.adoc[Lake table output] for
Delta Lake and Apache Iceberg tables
*
xref:pipeline/pipeline-run-configurations/native-spark-pipeline-engine.adoc[Native
Spark pipeline engine]
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/spark-lake-table-input.adoc
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/spark-lake-table-input.adoc
index cffc18c66d..836a1bfbed 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/spark-lake-table-input.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/spark-lake-table-input.adoc
@@ -16,19 +16,26 @@ under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
-:description: Spark lake table input reads Delta Lake or Apache Iceberg tables
on the native Spark pipeline engine, with optional time travel and projection.
-:page-engines: Hop Engine=no, Single Threaded=no, Native Spark=native, Beam
Spark=no, Beam Flink=no, Beam Dataflow=no
-:page-marketplace: hop-engines-spark
+:description: Lake table input reads Apache Iceberg tables on the Hop engine
and Delta Lake or Apache Iceberg tables on the native Spark engine, with
optional time travel and projection.
+:page-engines: Hop Engine=yes, Single Threaded=no, Native Spark=native, Beam
Spark=no, Beam Flink=no, Beam Dataflow=no
+:page-marketplace: hop-tech-lakehouse
-= image:transforms/icons/spark-lake-table-input.svg[Spark lake table input
transform Icon, role="image-doc-icon"] Spark lake table input
+= image:transforms/icons/spark-lake-table-input.svg[Lake table input transform
Icon, role="image-doc-icon"] Lake table input
== Description
-*Spark lake table input* reads an open table format table
(https://delta.io/[Delta Lake] or https://iceberg.apache.org/[Apache Iceberg])
as a Spark SQL `Dataset` on the
xref:pipeline/spark/getting-started-with-native-spark.adoc[native Spark
pipeline engine].
+*Lake table input* reads an open table format table:
https://iceberg.apache.org/[Apache Iceberg] or https://delta.io/[Delta Lake].
It supports **PATH** and **TABLE** identifiers, optional field projection, and
**time travel** by version/snapshot or timestamp.
-This transform is **not** available on the local Hop engine or on Beam
engines. Connector JARs ship with the native Spark plugin; see
xref:pipeline/spark/lakehouse.adoc[Lakehouse tables on the native Spark engine].
+The transform runs on two engines:
+
+* On the *Hop engine* (local and remote) it reads Apache Iceberg tables with
the Iceberg Java library. All files are read through Hop VFS, so a table can
live anywhere Hop can reach: local disk, S3, MinIO, Azure, Google Cloud
Storage. See <<hop-engine>>.
+* On the xref:pipeline/spark/getting-started-with-native-spark.adoc[native
Spark pipeline engine] it reads Delta Lake and Iceberg tables as a Spark SQL
`Dataset`. Connector JARs ship with the native Spark plugin; see
xref:pipeline/spark/lakehouse.adoc[Lakehouse tables on the native Spark engine].
+
+It isn't available on Beam engines.
+
+The transform is part of the Lakehouse plugin (`hop-tech-lakehouse`). On the
native Spark engine the Native Spark engine plugin (`hop-engines-spark`) is
needed as well. Both can be installed with the marketplace when they aren't in
your Hop distribution.
== Options
@@ -52,8 +59,8 @@ This transform is **not** available on the local Hop engine
or on Beam engines.
|Table identifier
|Used when mode is TABLE. Multi-part name: `catalog.namespace.table`.
-|Spark Catalog metadata name
-|Optional Hop xref:metadata-types/spark-catalog.adoc[Spark Catalog] entry to
register for TABLE mode (Iceberg Hadoop/REST, …). Leave empty if the catalog is
already configured on the session.
+|Catalog metadata name
+|The xref:metadata-types/spark-catalog.adoc[lakehouse catalog] to use in TABLE
mode (Iceberg Hadoop/REST, …). On the Spark engine you can leave it empty if
the catalog is already configured on the session.
|Time travel type
|`NONE` (current), `VERSION`, or `TIMESTAMP`.
@@ -62,10 +69,10 @@ This transform is **not** available on the local Hop engine
or on Beam engines.
|Delta table version number or Iceberg snapshot id (when type is VERSION).
Supports variables.
|Time travel timestamp
-|Spark-parseable timestamp string when type is TIMESTAMP, e.g. `2024-01-15
10:30:00`.
+|Timestamp when type is TIMESTAMP, e.g. `2024-01-15 10:30:00`. The Hop engine
also accepts an ISO-8601 timestamp with an offset and epoch milliseconds.
|Extra options
-|Optional Spark read options as `key=value` lines (advanced).
+|Optional Spark read options as `key=value` lines (advanced). Not used on the
Hop engine.
|Fields
|Optional projection. When empty, all columns are returned. When set, columns
are selected/cast by name (same idea as Spark file input with a schema).
@@ -92,6 +99,24 @@ This transform is **not** available on the local Hop engine
or on Beam engines.
Tip: After two Overwrite writes, Delta version `0` is typically the first
commit. For Iceberg, inspect `+{table}.snapshots+` (or catalog metadata) for
snapshot ids.
+[[hop-engine]]
+== On the Hop engine
+
+On the local and remote Hop engines the transform reads Apache Iceberg tables.
Delta Lake tables give an error that asks you to use the Spark engine for now.
+
+*Finding the table*
+
+* *PATH* mode reads the table in the given folder. The folder can be any Hop
VFS location, for example `file:///data/orders` or `s3://lake/orders`. When the
table has a `metadata/version-hint.text` file (tables written by a Hadoop
catalog, by the Spark engine in PATH mode, or by Lake table output on the Hop
engine) the version it names is read. Otherwise the newest metadata file is
read.
+* *TABLE* mode looks the table up through the selected
xref:metadata-types/spark-catalog.adoc[lakehouse catalog]. The Hop engine
supports the `hadoop` and `rest` catalog types. A catalog name in front of the
identifier (`lake.db.orders`) is dropped, so the same identifier works on both
engines. With a REST catalog the files are read through Hop VFS, unless the
catalog's extra configuration names another `io-impl`, for example
`org.apache.iceberg.aws.s3.S3FileIO` for credentials vended by [...]
+
+*Reading*
+
+* Iceberg plans the read: it skips the files and row groups that can't match,
using the statistics in the table metadata.
+* Delete files written by other engines (position and equality deletes, for
example from Flink or Spark) are applied.
+* Without a field list all columns are returned, with these Hop types:
booleans as Boolean, integers as Integer, floating point numbers as Number,
decimals as BigNumber, dates as Date, timestamps as Timestamp, binary values as
Binary and everything else as String. Dates and timestamps without a time zone
(`date`, `timestamp`, `timestamp_ns`) are read as local dates and date-times in
the time zone of the Hop JVM, as Parquet Input does, so the wall-clock value in
the table is kept. Timesta [...]
+* When the transform runs as several copies, the copies share the table
between them. Each part of a data file is read by exactly one copy.
+* A time travel timestamp without a time zone is in the time zone of the Hop
server.
+
== Metadata injection
All options support xref:pipeline/metadata-injection.adoc[metadata injection]
(ETL Metadata Injection).
@@ -104,11 +129,11 @@ Typical pattern: template pipeline with Lake Table Input;
injecting pipeline sup
== Notes
-* Prefer Lake Table Input over *Spark file input* for `delta` / `iceberg` so
session extensions, Iceberg PATH catalogs, and probes stay consistent.
+* On the Spark engine, prefer Lake table input over *Spark file input* for
`delta` / `iceberg` so session extensions, Iceberg PATH catalogs, and probes
stay consistent.
* See the xref:pipeline/spark/lakehouse.adoc[lakehouse guide] for packaging,
PATH vs TABLE, and troubleshooting.
== Related pages
* xref:pipeline/spark/lakehouse.adoc[Lakehouse tables on the native Spark
engine]
-* xref:pipeline/transforms/spark-lake-table-output.adoc[Spark lake table
output]
-* xref:metadata-types/spark-catalog.adoc[Spark Catalog]
+* xref:pipeline/transforms/spark-lake-table-output.adoc[Lake table output]
+* xref:metadata-types/spark-catalog.adoc[Lakehouse catalog]
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/spark-lake-table-maintenance.adoc
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/spark-lake-table-maintenance.adoc
index 97029fc01f..f786fb5246 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/spark-lake-table-maintenance.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/spark-lake-table-maintenance.adoc
@@ -45,7 +45,7 @@ Connectors ship with the native Spark plugin. Session rules:
xref:pipeline/spark
|Identifier mode
|`PATH` or `TABLE` for the target table.
-|Table path / Table identifier / Spark Catalog
+|Table path / Table identifier / Lakehouse catalog
|Same semantics as other Lake Table transforms.
|Operation
@@ -120,5 +120,5 @@ Injecting pipelines can drive fleet-wide OPTIMIZE/VACUUM
across many tables; sti
== Related pages
* xref:pipeline/spark/lakehouse.adoc[Lakehouse tables on the native Spark
engine]
-* xref:pipeline/transforms/spark-lake-table-output.adoc[Spark lake table
output]
+* xref:pipeline/transforms/spark-lake-table-output.adoc[Lake table output]
* xref:pipeline/transforms/spark-lake-table-merge.adoc[Spark lake table merge]
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/spark-lake-table-merge.adoc
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/spark-lake-table-merge.adoc
index a654871ab5..b7e0c31d5f 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/spark-lake-table-merge.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/spark-lake-table-merge.adoc
@@ -45,7 +45,7 @@ Connectors ship with the native Spark plugin. Session rules:
xref:pipeline/spark
|Identifier mode
|`PATH` or `TABLE` for the **target** table.
-|Table path / Table identifier / Spark Catalog
+|Table path / Table identifier / Lakehouse catalog
|Same semantics as Lake Table Input/Output for the merge target.
|Merge condition
@@ -99,5 +99,5 @@ Useful for multi-table CDC/upsert templates where merge keys
and target tables v
== Related pages
* xref:pipeline/spark/lakehouse.adoc[Lakehouse tables on the native Spark
engine]
-* xref:pipeline/transforms/spark-lake-table-output.adoc[Spark lake table
output]
-* xref:pipeline/transforms/spark-lake-table-input.adoc[Spark lake table input]
+* xref:pipeline/transforms/spark-lake-table-output.adoc[Lake table output]
+* xref:pipeline/transforms/spark-lake-table-input.adoc[Lake table input]
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/spark-lake-table-output.adoc
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/spark-lake-table-output.adoc
index 683fdd6789..e40587fc06 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/spark-lake-table-output.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/spark-lake-table-output.adoc
@@ -16,19 +16,24 @@ under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
-:description: Spark lake table output writes Delta Lake or Apache Iceberg
tables on the native Spark pipeline engine with ACID commit semantics.
-:page-engines: Hop Engine=no, Single Threaded=no, Native Spark=native, Beam
Spark=no, Beam Flink=no, Beam Dataflow=no
-:page-marketplace: hop-engines-spark
+:description: Lake table output writes Apache Iceberg tables on the Hop engine
and Delta Lake or Apache Iceberg tables on the native Spark engine, with ACID
commit semantics.
+:page-engines: Hop Engine=yes, Single Threaded=no, Native Spark=native, Beam
Spark=no, Beam Flink=no, Beam Dataflow=no
+:page-marketplace: hop-tech-lakehouse
-= image:transforms/icons/spark-lake-table-output.svg[Spark lake table output
transform Icon, role="image-doc-icon"] Spark lake table output
+= image:transforms/icons/spark-lake-table-output.svg[Lake table output
transform Icon, role="image-doc-icon"] Lake table output
== Description
-*Spark lake table output* writes the incoming Spark `Dataset` to a
https://delta.io/[Delta Lake] or https://iceberg.apache.org/[Apache Iceberg]
table using table-format commit semantics on the
xref:pipeline/spark/getting-started-with-native-spark.adoc[native Spark
pipeline engine].
+*Lake table output* writes the incoming rows to an
https://iceberg.apache.org/[Apache Iceberg] or https://delta.io/[Delta Lake]
table, using the table format's commits so readers never see a half-written run.
-Unlike *Spark file output* (classic file formats), the **default save mode is
ErrorIfExists** so ACID tables are not overwritten by accident.
+The **default save mode is ErrorIfExists**, so an existing table isn't
overwritten by accident.
-This transform is **native Spark only**. Connectors ship with the native Spark
plugin; see xref:pipeline/spark/lakehouse.adoc[Lakehouse tables on the native
Spark engine].
+The transform runs on two engines:
+
+* On the *Hop engine* (local and remote) it writes Apache Iceberg tables with
the Iceberg Java library, through Hop VFS, and passes the rows on unchanged.
See <<hop-engine>>.
+* On the xref:pipeline/spark/getting-started-with-native-spark.adoc[native
Spark pipeline engine] it writes the incoming `Dataset` to a Delta Lake or
Iceberg table. Connectors ship with the native Spark plugin; see
xref:pipeline/spark/lakehouse.adoc[Lakehouse tables on the native Spark engine].
+
+The transform is part of the Lakehouse plugin (`hop-tech-lakehouse`). On the
native Spark engine the Native Spark engine plugin (`hop-engines-spark`) is
needed as well. Both can be installed with the marketplace when they aren't in
your Hop distribution.
== Options
@@ -51,8 +56,8 @@ This transform is **native Spark only**. Connectors ship with
the native Spark p
|Table identifier
|TABLE mode: e.g. `lake.db.orders`.
-|Spark Catalog metadata name
-|Optional xref:metadata-types/spark-catalog.adoc[Spark Catalog] Hop metadata
for TABLE mode.
+|Catalog metadata name
+|The xref:metadata-types/spark-catalog.adoc[lakehouse catalog] for TABLE mode.
Optional on the Spark engine.
|Save mode
|`ErrorIfExists` (**default**), `Append`, `Overwrite`, or `Ignore`. Choose
Overwrite or Append explicitly when you intend to replace or extend data.
@@ -61,13 +66,62 @@ This transform is **native Spark only**. Connectors ship
with the native Spark p
|Optional comma-separated partition column list.
|Coalesce
-|Optional partition count before write (e.g. `1` for a single file layout —
costs a shuffle).
+|Optional partition count before write (e.g. `1` for a single file layout —
costs a shuffle). Spark engine only.
|Extra options
-|Optional Spark write options as `key=value` lines (advanced).
+|Optional Spark write options as `key=value` lines (advanced). Spark engine
only.
|===
-== Behaviour
+[[hop-engine]]
+== On the Hop engine
+
+On the local and remote Hop engines the transform writes Apache Iceberg
tables. Delta Lake tables give an error that asks you to use the Spark engine
for now.
+
+*One commit per pipeline run*
+
+Every copy of the transform writes its own Parquet files. Nothing is visible
in the table until the whole pipeline has finished:
+
+* When the pipeline ends without errors, all files of the run are committed as
one Iceberg snapshot. The snapshot summary records the pipeline name
(`hop.pipeline`) and the run's log channel id (`hop.run-id`).
+* When the pipeline fails or is stopped, the files are deleted and the table
is left as it was. This includes a failure in a transform *after* Lake table
output: the run is rolled back, like a database transaction.
+
+*Save modes*
+
+[options="header",cols="1,2,2"]
+|===
+|Save mode |Table exists |Table doesn't exist
+
+|ErrorIfExists
+|The pipeline fails, the table is left alone
+|The table is created
+
+|Append
+|The rows are added
+|The table is created
+
+|Overwrite
+|All rows of the table are replaced by the rows of this run. The run fails
instead if another writer changed the table since the run started.
+|The table is created
+
+|Ignore
+|Nothing is written; the rows are still passed on
+|The table is created
+|===
+
+A new table gets its columns from the incoming rows (Integer as `long`, Number
as `double`, BigNumber as `decimal`, String as `string`, Boolean as `boolean`,
Date and Timestamp as `timestamp` without time zone, Binary as `binary`) and
identity partitions on the *Partition by* columns. It is created in the same
commit as its first rows, so a failed run doesn't leave an empty table behind.
When the table exists, the rows are matched to its columns by name.
+
+* A BigNumber column gets the field's length as precision (at most 38) and its
precision as scale when the length is greater than 0; otherwise it is
`decimal(38, 10)`.
+* Dates and timestamps are written as the wall-clock time in the time zone of
the Hop JVM, so they read back unchanged on the Hop engine, as with Parquet
Output. Into an existing `timestamptz` column they are written as instants.
+
+Unlike the Spark engine, Overwrite on the Hop engine keeps the table's schema
and partitioning and only replaces its rows.
+
+*Where the table is*
+
+* *PATH* mode writes the table in the given Hop VFS folder, in the layout of
Hadoop catalogs and of the Spark engine's PATH mode
(`metadata/v<N>.metadata.json` plus `version-hint.text`). The Spark and Hop
engines can read and write the same table. A folder that holds a table managed
by a catalog is refused: write that table through its catalog in TABLE mode.
+* *TABLE* mode writes through the selected lakehouse catalog. The Hop engine
supports the `hadoop` and `rest` catalog types.
+
+On a local file system (`file://`), a PATH table commit publishes the new
metadata file with an operation that fails if another writer already committed
that version: a hard link, or an exclusive create where hard links aren't
supported. Several pipelines can write the same table there; of two conflicting
commits one fails and is retried. On HDFS, object stores like S3 and other
non-local Hop VFS locations, the commit checks that the version is free and
then writes it, which isn't atomic [...]
+
+== On the Spark engine
* The write runs as a Spark **action** when the engine materialises the graph.
* An empty leaf Dataset is registered so a later engine-wide `count()` does
not re-write the table.
@@ -87,11 +141,11 @@ Use this with a template write path when landing many
tables (different paths/id
== Notes
* Requires exactly one main-stream upstream hop (the data to write).
-* For upserts, use xref:pipeline/transforms/spark-lake-table-merge.adoc[Spark
lake table merge] instead of Overwrite.
+* For upserts on the Spark engine, use
xref:pipeline/transforms/spark-lake-table-merge.adoc[Spark lake table merge]
instead of Overwrite.
* Packaging, session conf, and troubleshooting:
xref:pipeline/spark/lakehouse.adoc[lakehouse guide].
== Related pages
* xref:pipeline/spark/lakehouse.adoc[Lakehouse tables on the native Spark
engine]
-* xref:pipeline/transforms/spark-lake-table-input.adoc[Spark lake table input]
+* xref:pipeline/transforms/spark-lake-table-input.adoc[Lake table input]
* xref:pipeline/transforms/spark-lake-table-merge.adoc[Spark lake table merge]
diff --git a/integration-tests/iceberg/0004-read-local.hpl
b/integration-tests/iceberg/0004-read-local.hpl
new file mode 100644
index 0000000000..b0f7fa1295
--- /dev/null
+++ b/integration-tests/iceberg/0004-read-local.hpl
@@ -0,0 +1,154 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<pipeline>
+ <info>
+ <name>0004-read-local</name>
+ <name_sync_with_filename>Y</name_sync_with_filename>
+ <description>iceberg PATH read + CSV extract</description>
+ <pipeline_type>Normal</pipeline_type>
+ <parameters/>
+ <capture_transform_performance>N</capture_transform_performance>
+ <created_user>-</created_user>
+ <created_date>2026/07/16 00:00:00.000</created_date>
+ <modified_user>-</modified_user>
+ <modified_date>2026/07/16 00:00:00.000</modified_date>
+ </info>
+ <notepads/>
+ <order>
+ <hop>
+ <from>Lake Input</from>
+ <to>CSV Extract</to>
+ <enabled>Y</enabled>
+ </hop>
+ </order>
+
+ <transform>
+ <type>SparkLakeTableInput</type>
+ <name>Lake Input</name>
+ <description/>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <format>iceberg</format>
+ <identifier_mode>PATH</identifier_mode>
+ <table_path>file:///tmp/hop-it-iceberg/0004-table</table_path>
+ <table_identifier/>
+ <catalog_metadata_name/>
+ <time_travel_type>NONE</time_travel_type>
+ <time_travel_version></time_travel_version>
+ <time_travel_timestamp></time_travel_timestamp>
+ <extra_options/>
+ <fields>
+ <field>
+ <name>id</name>
+ <type>Integer</type>
+ <length>-1</length>
+ <precision>0</precision>
+ </field>
+ <field>
+ <name>name</name>
+ <type>String</type>
+ <length>-1</length>
+ <precision>-1</precision>
+ </field>
+ </fields>
+ <attributes/>
+ <GUI>
+ <xloc>160</xloc>
+ <yloc>128</yloc>
+ </GUI>
+ </transform>
+ <transform>
+ <name>CSV Extract</name>
+ <type>TextFileOutput</type>
+ <description/>
+ <distribute>Y</distribute>
+ <custom_distribution/>
+ <copies>1</copies>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <separator>,</separator>
+ <enclosure>"</enclosure>
+ <enclosure_forced>N</enclosure_forced>
+ <enclosure_fix_disabled>N</enclosure_fix_disabled>
+ <header>Y</header>
+ <footer>N</footer>
+ <format>UNIX</format>
+ <compression>None</compression>
+ <encoding>UTF-8</encoding>
+ <endedLine/>
+ <fileNameInField>N</fileNameInField>
+ <fileNameField/>
+ <create_parent_folder>Y</create_parent_folder>
+ <file>
+ <name>/tmp/hop-it-iceberg/0004-extract/part</name>
+ <servlet_output>N</servlet_output>
+ <do_not_open_new_file_init>N</do_not_open_new_file_init>
+ <extention>csv</extention>
+ <append>N</append>
+ <split>N</split>
+ <haspartno>N</haspartno>
+ <add_date>N</add_date>
+ <add_time>N</add_time>
+ <SpecifyFormat>N</SpecifyFormat>
+ <date_time_format/>
+ <add_to_result_filenames>N</add_to_result_filenames>
+ <pad>N</pad>
+ <fast_dump>N</fast_dump>
+ <splitevery>0</splitevery>
+ </file>
+ <fields>
+ <field>
+ <name>id</name>
+ <type>Integer</type>
+ <format>#</format>
+ <currency/>
+ <decimal/>
+ <group/>
+ <nullif/>
+ <trim_type>none</trim_type>
+ <length>-1</length>
+ <precision>-1</precision>
+ </field>
+ <field>
+ <name>name</name>
+ <type>String</type>
+ <format/>
+ <currency/>
+ <decimal/>
+ <group/>
+ <nullif/>
+ <trim_type>none</trim_type>
+ <length>-1</length>
+ <precision>-1</precision>
+ </field>
+ </fields>
+ <attributes/>
+ <GUI>
+ <xloc>480</xloc>
+ <yloc>128</yloc>
+ </GUI>
+ </transform>
+ <attributes/>
+ <transform_error_handling/>
+</pipeline>
diff --git a/integration-tests/iceberg/0004-validation.hpl
b/integration-tests/iceberg/0004-validation.hpl
new file mode 100644
index 0000000000..e32f9b2239
--- /dev/null
+++ b/integration-tests/iceberg/0004-validation.hpl
@@ -0,0 +1,167 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<pipeline>
+ <info>
+ <name>0004-validation</name>
+ <name_sync_with_filename>Y</name_sync_with_filename>
+ <description>Validate 0004 extract against golden</description>
+ <pipeline_type>Normal</pipeline_type>
+ <parameters/>
+ <capture_transform_performance>N</capture_transform_performance>
+ <created_user>-</created_user>
+ <created_date>2026/07/16 00:00:00.000</created_date>
+ <modified_user>-</modified_user>
+ <modified_date>2026/07/16 00:00:00.000</modified_date>
+ </info>
+ <notepads/>
+ <order>
+ <hop>
+ <from>Read extract CSV</from>
+ <to>Verify</to>
+ <enabled>Y</enabled>
+ </hop>
+ </order>
+ <transform>
+ <name>Read extract CSV</name>
+ <type>TextFileInput2</type>
+ <description/>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <accept_filenames>N</accept_filenames>
+ <passing_through_fields>N</passing_through_fields>
+ <accept_field/>
+ <accept_transform_name/>
+ <separator>,</separator>
+ <enclosure>"</enclosure>
+ <enclosure_breaks>N</enclosure_breaks>
+ <escapechar/>
+ <header>Y</header>
+ <nr_headerlines>1</nr_headerlines>
+ <footer>N</footer>
+ <nr_footerlines>1</nr_footerlines>
+ <line_wrapped>N</line_wrapped>
+ <nr_wraps>1</nr_wraps>
+ <layout_paged>N</layout_paged>
+ <nr_lines_per_page>80</nr_lines_per_page>
+ <nr_lines_doc_header>0</nr_lines_doc_header>
+ <noempty>Y</noempty>
+ <include>N</include>
+ <include_field/>
+ <rownum>N</rownum>
+ <rownumByFile>N</rownumByFile>
+ <rownum_field/>
+ <format>Unix</format>
+ <encoding>UTF-8</encoding>
+ <length>Characters</length>
+ <add_to_result_filenames>N</add_to_result_filenames>
+ <file>
+ <name>/tmp/hop-it-iceberg/0004-extract/</name>
+ <filemask>.*\.csv</filemask>
+ <exclude_filemask/>
+ <file_required>Y</file_required>
+ <include_subfolders>N</include_subfolders>
+ <type>CSV</type>
+ <compression>None</compression>
+ </file>
+ <filters/>
+ <fields>
+ <field>
+ <name>id</name>
+ <type>Integer</type>
+ <format>#</format>
+ <currency/>
+ <decimal/>
+ <group/>
+ <nullif/>
+ <ifnull/>
+ <position>-1</position>
+ <length>-1</length>
+ <precision>0</precision>
+ <trim_type>both</trim_type>
+ <repeat>N</repeat>
+ </field>
+ <field>
+ <name>name</name>
+ <type>String</type>
+ <format/>
+ <currency/>
+ <decimal/>
+ <group/>
+ <nullif/>
+ <ifnull/>
+ <position>-1</position>
+ <length>-1</length>
+ <precision>-1</precision>
+ <trim_type>both</trim_type>
+ <repeat>N</repeat>
+ </field>
+ </fields>
+ <limit>0</limit>
+ <error_ignored>N</error_ignored>
+ <skip_bad_files>N</skip_bad_files>
+ <file_error_field/>
+ <file_error_message_field/>
+ <error_line_skipped>N</error_line_skipped>
+ <error_count_field/>
+ <error_fields_field/>
+ <error_text_field/>
+ <bad_line_files_destination_directory/>
+ <bad_line_files_extension>warning</bad_line_files_extension>
+ <error_line_files_destination_directory/>
+ <error_line_files_extension>error</error_line_files_extension>
+ <line_number_files_destination_directory/>
+ <line_number_files_extension>line</line_number_files_extension>
+ <date_format_lenient>Y</date_format_lenient>
+ <date_format_locale>en_US</date_format_locale>
+ <shortFileFieldName/>
+ <pathFieldName/>
+ <hiddenFieldName/>
+ <lastModificationTimeFieldName/>
+ <uriNameFieldName/>
+ <rootUriNameFieldName/>
+ <extensionFieldName/>
+ <sizeFieldName/>
+ <attributes/>
+ <GUI>
+ <xloc>160</xloc>
+ <yloc>128</yloc>
+ </GUI>
+ </transform>
+ <transform>
+ <name>Verify</name>
+ <type>Dummy</type>
+ <description/>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <attributes/>
+ <GUI>
+ <xloc>400</xloc>
+ <yloc>128</yloc>
+ </GUI>
+ </transform>
+ <attributes/>
+ <transform_error_handling/>
+</pipeline>
diff --git a/integration-tests/iceberg/0004-write-local.hpl
b/integration-tests/iceberg/0004-write-local.hpl
new file mode 100644
index 0000000000..5f5cd9cdde
--- /dev/null
+++ b/integration-tests/iceberg/0004-write-local.hpl
@@ -0,0 +1,120 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<pipeline>
+ <info>
+ <name>0004-write-local</name>
+ <name_sync_with_filename>Y</name_sync_with_filename>
+ <description>iceberg PATH write 10 rows</description>
+ <pipeline_type>Normal</pipeline_type>
+ <parameters/>
+ <capture_transform_performance>N</capture_transform_performance>
+ <created_user>-</created_user>
+ <created_date>2026/07/16 00:00:00.000</created_date>
+ <modified_user>-</modified_user>
+ <modified_date>2026/07/16 00:00:00.000</modified_date>
+ </info>
+ <notepads/>
+ <order>
+ <hop>
+ <from>Seed</from>
+ <to>Lake Output</to>
+ <enabled>Y</enabled>
+ </hop>
+ </order>
+
+ <transform>
+ <name>Seed</name>
+ <type>DataGrid</type>
+ <description/>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <data>
+ <line><item>0</item><item>n0</item></line>
+ <line><item>1</item><item>n1</item></line>
+ <line><item>2</item><item>n2</item></line>
+ <line><item>3</item><item>n3</item></line>
+ <line><item>4</item><item>n4</item></line>
+ <line><item>5</item><item>n5</item></line>
+ <line><item>6</item><item>n6</item></line>
+ <line><item>7</item><item>n7</item></line>
+ <line><item>8</item><item>n8</item></line>
+ <line><item>9</item><item>n9</item></line>
+ </data>
+
+ <fields>
+ <field>
+ <name>id</name>
+ <type>Integer</type>
+ <format>#</format>
+ <length>-1</length>
+ <precision>0</precision>
+ <currency/>
+ <decimal/>
+ <group/>
+ <set_empty_string>N</set_empty_string>
+ </field>
+ <field>
+ <name>name</name>
+ <type>String</type>
+ <format/>
+ <length>-1</length>
+ <precision>-1</precision>
+ <currency/>
+ <decimal/>
+ <group/>
+ <set_empty_string>N</set_empty_string>
+ </field>
+ </fields>
+ <attributes/>
+ <GUI>
+ <xloc>96</xloc>
+ <yloc>128</yloc>
+ </GUI>
+ </transform>
+ <transform>
+ <type>SparkLakeTableOutput</type>
+ <name>Lake Output</name>
+ <description/>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <format>iceberg</format>
+ <identifier_mode>PATH</identifier_mode>
+ <table_path>file:///tmp/hop-it-iceberg/0004-table</table_path>
+ <table_identifier/>
+ <catalog_metadata_name/>
+ <save_mode>Overwrite</save_mode>
+ <partition_by/>
+ <coalesce/>
+ <extra_options/>
+ <attributes/>
+ <GUI>
+ <xloc>320</xloc>
+ <yloc>128</yloc>
+ </GUI>
+ </transform>
+ <attributes/>
+ <transform_error_handling/>
+</pipeline>
diff --git a/integration-tests/iceberg/README.md
b/integration-tests/iceberg/README.md
index 92f9e81bfa..f244e6dd51 100644
--- a/integration-tests/iceberg/README.md
+++ b/integration-tests/iceberg/README.md
@@ -17,8 +17,8 @@ limitations under the License.
# iceberg integration tests (iceberg)
-End-to-end Hop IT for **Spark Lake Table** transforms on the native Spark
engine
-using **iceberg** PATH mode.
+End-to-end Hop IT for the **Lake table** transforms with **iceberg** PATH
tables, on the
+native Spark engine and on the local Hop engine.
## Prerequisites
@@ -33,6 +33,7 @@ using **iceberg** PATH mode.
| `main-0001-input-output` | Data Grid → Lake Output → Lake Input → CSV →
golden (10 rows) |
| `main-0002-overwrite-timetravel` | Write 5, overwrite 20, read current (and
time travel where applicable) |
| `main-0003-merge` | Seed (1,a)(2,b), MERGE update/insert → golden 3 rows |
+| `main-0004-local-input-output` | Same round trip as 0001 on the local
engine: Data Grid → Lake Output → Lake Input → Text file output → golden (10
rows) |
0002 validates double Overwrite + current read. Iceberg snapshot-id time
travel is covered in Java unit tests (`SparkLakeTableTimeTravelTest`).
diff --git a/integration-tests/iceberg/main-0004-local-input-output.hwf
b/integration-tests/iceberg/main-0004-local-input-output.hwf
new file mode 100644
index 0000000000..4eea4a60e8
--- /dev/null
+++ b/integration-tests/iceberg/main-0004-local-input-output.hwf
@@ -0,0 +1,180 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<workflow>
+ <name>main-0004-local-input-output</name>
+ <name_sync_with_filename>Y</name_sync_with_filename>
+ <description>Lake Table Input/Output PATH round-trip on the local
engine</description>
+ <extended_description/>
+ <workflow_version/>
+ <created_user>-</created_user>
+ <created_date>2026/10/07 00:00:00.000</created_date>
+ <modified_user>-</modified_user>
+ <modified_date>2026/10/07 00:00:00.000</modified_date>
+ <parameters/>
+ <actions>
+ <action>
+ <name>Start</name>
+ <description/>
+ <type>SPECIAL</type>
+ <attributes/>
+ <repeat>N</repeat>
+ <schedulerType>0</schedulerType>
+ <intervalSeconds>0</intervalSeconds>
+ <intervalMinutes>60</intervalMinutes>
+ <hour>12</hour>
+ <minutes>0</minutes>
+ <weekDay>1</weekDay>
+ <DayOfMonth>1</DayOfMonth>
+ <parallel>N</parallel>
+ <xloc>64</xloc>
+ <yloc>96</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>clean work dir</name>
+ <description>Remove previous table/extract data under
/tmp/hop-it-iceberg (always succeeds)</description>
+ <type>SHELL</type>
+ <attributes/>
+ <add_date>N</add_date>
+ <add_time>N</add_time>
+ <arg_from_previous>N</arg_from_previous>
+ <arguments>
+</arguments>
+ <exec_per_row>N</exec_per_row>
+ <insertScript>Y</insertScript>
+ <loglevel>Basic</loglevel>
+ <script>#!/bin/bash
+set +e
+WORK="/tmp/hop-it-iceberg"
+echo "Cleaning IT work dir: $WORK"
+rm -rf "$WORK"
+mkdir -p "$WORK"
+exit 0
+</script>
+ <set_append_logfile>N</set_append_logfile>
+ <set_logfile>N</set_logfile>
+ <parallel>N</parallel>
+ <xloc>224</xloc>
+ <yloc>96</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>0004-write-local</name>
+ <description/>
+ <type>PIPELINE</type>
+ <attributes/>
+ <filename>${PROJECT_HOME}/0004-write-local.hpl</filename>
+ <params_from_previous>N</params_from_previous>
+ <exec_per_row>N</exec_per_row>
+ <clear_rows>N</clear_rows>
+ <clear_files>N</clear_files>
+ <set_logfile>N</set_logfile>
+ <logfile/>
+ <logext/>
+ <add_date>N</add_date>
+ <add_time>N</add_time>
+ <loglevel>Basic</loglevel>
+ <set_append_logfile>N</set_append_logfile>
+ <wait_until_finished>Y</wait_until_finished>
+ <create_parent_folder>Y</create_parent_folder>
+ <run_configuration>local</run_configuration>
+ <parameters>
+ <pass_all_parameters>Y</pass_all_parameters>
+ </parameters>
+ <parallel>N</parallel>
+ <xloc>400</xloc>
+ <yloc>96</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>0004-read-local</name>
+ <description/>
+ <type>PIPELINE</type>
+ <attributes/>
+ <filename>${PROJECT_HOME}/0004-read-local.hpl</filename>
+ <params_from_previous>N</params_from_previous>
+ <exec_per_row>N</exec_per_row>
+ <clear_rows>N</clear_rows>
+ <clear_files>N</clear_files>
+ <set_logfile>N</set_logfile>
+ <logfile/>
+ <logext/>
+ <add_date>N</add_date>
+ <add_time>N</add_time>
+ <loglevel>Basic</loglevel>
+ <set_append_logfile>N</set_append_logfile>
+ <wait_until_finished>Y</wait_until_finished>
+ <create_parent_folder>Y</create_parent_folder>
+ <run_configuration>local</run_configuration>
+ <parameters>
+ <pass_all_parameters>Y</pass_all_parameters>
+ </parameters>
+ <parallel>N</parallel>
+ <xloc>576</xloc>
+ <yloc>96</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>Run unit tests</name>
+ <description/>
+ <type>RunPipelineTests</type>
+ <attributes/>
+ <test_names>
+ <test_name>
+ <name>0004-validation UNIT</name>
+ </test_name>
+ </test_names>
+ <parallel>N</parallel>
+ <xloc>752</xloc>
+ <yloc>96</yloc>
+ <attributes_hac/>
+ </action>
+ </actions>
+ <hops>
+ <hop>
+ <from>Start</from>
+ <to>clean work dir</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>Y</unconditional>
+ </hop>
+ <hop>
+ <from>clean work dir</from>
+ <to>0004-write-local</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>0004-write-local</from>
+ <to>0004-read-local</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>0004-read-local</from>
+ <to>Run unit tests</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ </hops>
+ <notepads/>
+ <attributes/>
+</workflow>
diff --git a/integration-tests/iceberg/metadata/unit-test/0004-validation
UNIT.json b/integration-tests/iceberg/metadata/unit-test/0004-validation
UNIT.json
new file mode 100644
index 0000000000..3593a86ace
--- /dev/null
+++ b/integration-tests/iceberg/metadata/unit-test/0004-validation UNIT.json
@@ -0,0 +1,32 @@
+{
+ "variableValues": [],
+ "database_replacements": [],
+ "autoOpening": true,
+ "basePath": "",
+ "golden_data_sets": [
+ {
+ "field_mappings": [
+ {
+ "transform_field": "id",
+ "data_set_field": "id"
+ },
+ {
+ "transform_field": "name",
+ "data_set_field": "name"
+ }
+ ],
+ "field_order": [
+ "id"
+ ],
+ "data_set_name": "0001-roundtrip-golden",
+ "transform_name": "Verify"
+ }
+ ],
+ "input_data_sets": [],
+ "name": "0004-validation UNIT",
+ "description": "Golden compare for 0004-validation",
+ "persist_filename": "",
+ "trans_test_tweaks": [],
+ "pipeline_filename": "./0004-validation.hpl",
+ "test_type": "UNIT_TEST"
+}
diff --git
a/plugins/misc/marketplace/src/test/java/org/apache/hop/marketplace/catalog/OptionalPluginCatalogTest.java
b/plugins/misc/marketplace/src/test/java/org/apache/hop/marketplace/catalog/OptionalPluginCatalogTest.java
index 15ceb20090..32adfd3576 100644
---
a/plugins/misc/marketplace/src/test/java/org/apache/hop/marketplace/catalog/OptionalPluginCatalogTest.java
+++
b/plugins/misc/marketplace/src/test/java/org/apache/hop/marketplace/catalog/OptionalPluginCatalogTest.java
@@ -33,6 +33,7 @@ class OptionalPluginCatalogTest {
List<String> ids = OptionalPluginCatalog.artifactIds();
assertTrue(ids.contains("hop-tech-parquet"));
assertTrue(ids.contains("hop-engines-spark"));
+ assertTrue(ids.contains("hop-tech-lakehouse"));
assertTrue(ids.contains("hop-engines-beam"));
assertTrue(ids.contains("hop-transform-edi2xml"));
}
diff --git
a/plugins/tech/lakehouse/src/main/resources/org/apache/hop/lakehouse/metadata/messages/messages_en_US.properties
b/plugins/tech/lakehouse/src/main/resources/org/apache/hop/lakehouse/metadata/messages/messages_en_US.properties
index 374326f0c7..fedeb433e5 100644
---
a/plugins/tech/lakehouse/src/main/resources/org/apache/hop/lakehouse/metadata/messages/messages_en_US.properties
+++
b/plugins/tech/lakehouse/src/main/resources/org/apache/hop/lakehouse/metadata/messages/messages_en_US.properties
@@ -15,8 +15,8 @@
# limitations under the License.
#
-SparkCatalog.Name=Spark Catalog
-SparkCatalog.Description=Spark SQL catalog for open table formats (Iceberg
Hadoop/REST, custom)
+SparkCatalog.Name=Lakehouse catalog
+SparkCatalog.Description=Catalog for open table formats (Iceberg Hadoop/REST,
custom), used by the lake table transforms
SparkCatalog.Name.Label=Name
SparkCatalog.CatalogName.Label=Spark catalog name
SparkCatalog.CatalogName.Tooltip=Name used in spark.sql.catalog.<name> (e.g.
lake). Must match the first part of table identifiers.
diff --git
a/plugins/tech/lakehouse/src/main/resources/org/apache/hop/lakehouse/transforms/messages/messages_en_US.properties
b/plugins/tech/lakehouse/src/main/resources/org/apache/hop/lakehouse/transforms/messages/messages_en_US.properties
index 6e2b521542..5bd32b7a64 100644
---
a/plugins/tech/lakehouse/src/main/resources/org/apache/hop/lakehouse/transforms/messages/messages_en_US.properties
+++
b/plugins/tech/lakehouse/src/main/resources/org/apache/hop/lakehouse/transforms/messages/messages_en_US.properties
@@ -15,22 +15,22 @@
# limitations under the License.
#
-SparkLakeTableInput.Name=Spark lake table input
-SparkLakeTableInput.Description=Read an open table format (Delta Lake / Apache
Iceberg) with the native Spark engine
+SparkLakeTableInput.Name=Lake table input
+SparkLakeTableInput.Description=Read an Apache Iceberg table, or a Delta Lake
table on the native Spark engine
SparkLakeTableInput.Keyword=spark,delta,iceberg,lake,table,input,acid
SparkLakeTableInput.Injection.Group.Fields=Fields (optional projection)
-SparkLakeTableOutput.Name=Spark lake table output
-SparkLakeTableOutput.Description=Write an open table format (Delta Lake /
Apache Iceberg) with the native Spark engine
+SparkLakeTableOutput.Name=Lake table output
+SparkLakeTableOutput.Description=Write an Apache Iceberg table, or a Delta
Lake table on the native Spark engine
SparkLakeTableOutput.Keyword=spark,delta,iceberg,lake,table,output,acid
-SparkLakeTableInputDialog.Shell.Title=Spark lake table input
+SparkLakeTableInputDialog.Shell.Title=Lake table input
SparkLakeTableInputDialog.Format=Table format
SparkLakeTableInputDialog.IdentifierMode=Identifier mode
-SparkLakeTableInputDialog.TablePath=Spark table path (PATH mode, Hadoop URI)
-SparkLakeTableInputDialog.TablePath.Tooltip=Lake table root as a Spark/Hadoop
URI (file\:///…, s3a\://…, …). Not a Hop VFS path (s3\://, named MinIO
connection schemes).
+SparkLakeTableInputDialog.TablePath=Table path (PATH mode)
+SparkLakeTableInputDialog.TablePath.Tooltip=Root folder of the table. On the
Hop engine any Hop VFS location (file\:///…, s3\://…). On the Spark engine a
Spark/Hadoop URI (file\:///…, s3a\://…), not a Hop VFS path.
SparkLakeTableInputDialog.TableIdentifier=Table identifier (TABLE mode, e.g.
lake.db.orders)
-SparkLakeTableInputDialog.CatalogMetadataName=Spark Catalog metadata name
(TABLE mode)
+SparkLakeTableInputDialog.CatalogMetadataName=Lakehouse catalog (TABLE mode)
SparkLakeTableInputDialog.TimeTravelType=Time travel
SparkLakeTableInputDialog.TimeTravelVersion=Version / snapshot id
SparkLakeTableInputDialog.TimeTravelTimestamp=As-of timestamp
@@ -41,13 +41,13 @@ SparkLakeTableInputDialog.Column.Type=Type
SparkLakeTableInputDialog.Column.Length=Length
SparkLakeTableInputDialog.Column.Precision=Precision
-SparkLakeTableOutputDialog.Shell.Title=Spark lake table output
+SparkLakeTableOutputDialog.Shell.Title=Lake table output
SparkLakeTableOutputDialog.Format=Table format
SparkLakeTableOutputDialog.IdentifierMode=Identifier mode
-SparkLakeTableOutputDialog.TablePath=Spark table path (PATH mode, Hadoop URI)
-SparkLakeTableOutputDialog.TablePath.Tooltip=Lake table root as a Spark/Hadoop
URI (file\:///…, s3a\://…, …). Not a Hop VFS path.
+SparkLakeTableOutputDialog.TablePath=Table path (PATH mode)
+SparkLakeTableOutputDialog.TablePath.Tooltip=Root folder of the table. On the
Hop engine any Hop VFS location (file\:///…, s3\://…). On the Spark engine a
Spark/Hadoop URI (file\:///…, s3a\://…), not a Hop VFS path.
SparkLakeTableOutputDialog.TableIdentifier=Table identifier (TABLE mode, e.g.
lake.db.orders)
-SparkLakeTableOutputDialog.CatalogMetadataName=Spark Catalog metadata name
(TABLE mode)
+SparkLakeTableOutputDialog.CatalogMetadataName=Lakehouse catalog (TABLE mode)
SparkLakeTableOutputDialog.SaveMode=Save mode
SparkLakeTableOutputDialog.PartitionBy=Partition by columns (comma-separated)
SparkLakeTableOutputDialog.Coalesce=Coalesce partitions (e.g. 1 for single
file)
@@ -63,7 +63,7 @@ SparkLakeTableMergeDialog.IdentifierMode=Identifier mode
SparkLakeTableMergeDialog.TablePath=Spark target table path (PATH mode, Hadoop
URI)
SparkLakeTableMergeDialog.TablePath.Tooltip=Target lake table root as a
Spark/Hadoop URI (file\:///…, s3a\://…, …). Not a Hop VFS path.
SparkLakeTableMergeDialog.TableIdentifier=Target table identifier (TABLE mode)
-SparkLakeTableMergeDialog.CatalogMetadataName=Spark Catalog metadata name
(TABLE mode)
+SparkLakeTableMergeDialog.CatalogMetadataName=Lakehouse catalog (TABLE mode)
SparkLakeTableMergeDialog.MergeCondition=Merge condition (ON clause, t=target
s=source)
SparkLakeTableMergeDialog.MatchedAction=When matched
SparkLakeTableMergeDialog.NotMatchedAction=When not matched
@@ -80,7 +80,7 @@ SparkLakeTableMaintenanceDialog.IdentifierMode=Identifier mode
SparkLakeTableMaintenanceDialog.TablePath=Spark table path (PATH mode, Hadoop
URI)
SparkLakeTableMaintenanceDialog.TablePath.Tooltip=Lake table root as a
Spark/Hadoop URI (file\:///…, s3a\://…, …). Not a Hop VFS path.
SparkLakeTableMaintenanceDialog.TableIdentifier=Table identifier (TABLE mode)
-SparkLakeTableMaintenanceDialog.CatalogMetadataName=Spark Catalog metadata
name (TABLE mode)
+SparkLakeTableMaintenanceDialog.CatalogMetadataName=Lakehouse catalog (TABLE
mode)
SparkLakeTableMaintenanceDialog.Operation=Operation
SparkLakeTableMaintenanceDialog.RetentionHours=Retention hours (VACUUM /
EXPIRE required)
SparkLakeTableMaintenanceDialog.RetainLast=Retain last N snapshots (Iceberg
expire)
