kotwal-itpro opened a new pull request, #8819: URL: https://github.com/apache/hop/pull/8819
Third step for #5524, on top of #8802 (now in main): **Lake table output writes Apache Iceberg tables on the local (and remote) engine**, without Spark. It uses the Iceberg Java library, with every file going through Hop VFS, and the same date and time handling #8802 settled on. ## Save modes The same as on the Spark engine: | Mode | Existing table | Missing table | |---|---|---| | Append | adds the rows | created | | Overwrite | replaces the rows | created | | ErrorIfExists | refused, the table is left alone | created | | Ignore | rows are passed on, nothing is written | created | A missing table is created from the incoming row layout, partitioned by the configured partition columns. ## One snapshot per pipeline run This is the part I'd most like a second pair of eyes on. - Every copy of the transform writes its own Parquet files. Nothing is committed when a copy finishes. - `CommitCoordinator` collects the files of all copies and commits them as **one snapshot when the whole pipeline has finished without errors**, from a pipeline-finished listener. If any transform failed, including one further down the pipeline after the output, the files are deleted and the table is unchanged. - A new table is created **in the same commit** as its first snapshot, so a failed run doesn't leave an empty table behind. - Overwrites validate from the snapshot the run started with (`validateFromSnapshot` plus `validateNoConflictingData` and `validateNoConflictingDeletes`), so a concurrent writer makes the commit fail instead of having its data silently replaced. - The snapshot summary records the pipeline name and a run id (`hop.pipeline`, `hop.run-id`), so a snapshot can be traced back to the run that wrote it. ## Path tables Path tables are written in the layout of Hadoop catalogs and Spark path tables (`metadata/v<N>.metadata.json` and `version-hint.text`), so the local engine and the Spark engine write the same table, and each can read what the other wrote. `PathTableOperations` commits by moving the new metadata file to the next version's name and fails if the name is taken. One limitation, the same one Iceberg documents for Hadoop tables: that move is atomic on local file systems and HDFS but not on object stores like S3, so a path table on S3 should have one writer at a time. The docs PR (next) will say so; tables that need concurrent writers should use a catalog. A path whose table belongs to a catalog (metadata files not named `v<N>`) is refused, because a commit there wouldn't be seen by the catalog. ## Testing - `hop-tech-lakehouse`: 76 tests. `LakeTableOutputLocalTest` runs real pipelines on the local engine: one snapshot from three copies, the types of a created table, partitioning, every save mode, a failure later in the pipeline both for a new table (no table created) and an existing one (table unchanged), writing through a hadoop catalog, refusing a catalog table by path, and writing timestamps while the JVM runs in Los Angeles time. - `hop-engines-spark`: 179 tests pass. - `apache-rat:check` and `spotless:apply` clean. ## Next 4. Docs (including the S3 single-writer note), display names, and a local engine integration test. I used an AI coding assistant while writing this change (noted in the commit with `Generated-by`); I reviewed and tested all of it. addresses #5524 - [x] I hereby declare this contribution to be licensed under the [Apache License Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0) 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
