kotwal-itpro opened a new pull request, #8802: URL: https://github.com/apache/hop/pull/8802
Second step for #5524. **This builds on #8792**, so until that is merged this PR also shows its commit. Only the last commit, "Read Iceberg tables with Lake table input on the local engine", is new here. Lake table input now reads Apache Iceberg tables on the local (and remote) engine, without Spark. It uses the Iceberg Java library, and every metadata, manifest and data file goes through Hop VFS, so tables can live anywhere Hop can reach with the VFS connections already configured. ## What it supports - **PATH mode** loads the table from its folder. It follows `version-hint.text` when there is one (Hadoop catalog tables and Spark path tables), otherwise it takes the newest `v<N>.metadata.json`. - **TABLE mode** uses the lakehouse catalog metadata object from #8792, with the `hadoop` and `rest` types for now. Other types give a clear error. - **Time travel** by snapshot id or by timestamp. - **Deletes:** position and equality delete files are applied. - **Field list:** the optional field list selects, orders and converts columns, the same as on Spark. Without it, all columns are read with their Iceberg types mapped to Hop types. - **Several copies** share the work. Each split goes to exactly one copy, chosen by a hash of its file path and offset. I first split by position, but Iceberg plans manifests in parallel, so splits don't come back in a stable order and two copies could read the same split. Delta tables give a clear error on these engines for now. The output, merge and maintenance transforms are unchanged and still Spark-only; local writes are the next PR. ## Two things worth a look - **Class loading.** Iceberg 1.11 registers its file formats in a static initializer that loads classes through the thread's context class loader. In unit tests that works; in an assembled client the transform thread's context class loader can't see the plugin's jars, so every read failed with `Format model is not registered for format PARQUET`. `PluginClassLoader` makes the plugin's class loader the context class loader around calls into Iceberg. I found this by running the pipelines in a built distribution, not in the tests. - **Spark plugin dependencies.** The Spark plugin now keeps only the lakehouse plugin jar on its compile classpath (the transitive dependencies are excluded), matching what it sees at runtime. Otherwise the unshaded Iceberg modules clash with `iceberg-spark-runtime` in the Spark tests. ## Testing - `hop-tech-lakehouse`: 41 tests. `LakeTableInputLocalTest` runs real pipelines on the local engine against tables written with the Iceberg library: all columns, copies reading every row exactly once, selected fields, time travel by id and timestamp, unknown snapshot, version hint, equality deletes, TABLE mode through a hadoop catalog object, and the Delta error. `IcebergTablesTest` and `HopVfsFileIOTest` cover table resolution, catalog properties and the VFS file IO. - `hop-engines-spark`: all 179 tests pass. - Also checked in an assembled client: a table written by the Spark engine and read back by this transform on the local engine. - `spotless:apply` and `apache-rat:check` clean. ## Next 3. Local engine write support for Iceberg (one atomic snapshot per pipeline run). 4. Docs, display names and a local engine integration test. I used an AI coding assistant while writing this change (noted in the commit with `Generated-by`); I reviewed and tested all of it. addresses #5524 - [x] I hereby declare this contribution to be licensed under the [Apache License Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0) 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
