kotwal-itpro commented on issue #5524: URL: https://github.com/apache/hop/issues/5524#issuecomment-6032313089
Thanks @bamaer, that's helpful. Agreed that full Iceberg integration is a much bigger job. Here's the starting point I have in mind, so people can comment before the first PR. **Scope of the first version (local engine, no Spark)** - **`Iceberg catalog` metadata type** (Connections): REST, Hive, Glue, JDBC and Hadoop catalogs, with templates in the same spirit as the Spark Catalog editor, a Test button that lists namespaces, and variables everywhere. - **`Iceberg table input`**: column selection with "Get fields", a filter grid that's pushed down into Iceberg's scan planning (so whole files get skipped), and time travel by snapshot, branch, tag or timestamp. - **`Iceberg table output`**: append, overwrite table, or overwrite matching partitions, plus optional create-if-missing with a partition spec, and "add missing columns" schema evolution. **A few design points** - **Storage goes through Hop VFS.** A small Iceberg `FileIO` on top of `HopVfs`, the same idea as `ParquetOutputFile`, so tables on S3, MinIO, Azure, GCS or local disk use the connections people already have. Catalogs that vend their own credentials (REST credential vending, S3 Tables) can switch to the catalog's FileIO instead. - **One atomic snapshot per pipeline run.** Every copy of the output transform writes its own data files, and the commit happens once, from a pipeline finished listener, and only if the whole pipeline ended without errors. If a transform further down fails or the run is stopped, nothing is committed and the written files are removed. Overwrites validate against the snapshot the run started from, so a concurrent writer causes a clear error instead of silently lost data. - **Parallel reads without coordination.** Each input copy keeps the scan splits whose hash of (file, offset) maps to its copy number. I first tried assigning by position in the task list, but Iceberg doesn't return tasks in a stable order between planning calls, so rows were read twice or skipped. Hashing fixed it. **Where it stands** I have a working prototype of the core, not the transforms or dialogs yet: the VFS FileIO, type mapping, the row writer, the commit coordinator and the reader. Its tests run against a JDBC catalog with every data and metadata file going through Hop VFS, and cover round trips, partitioned writes from several copies landing in one snapshot, filter pruning, an aborted run leaving no files behind, time travel, partition overwrite and a rejected concurrent overwrite. **Plan** I'd split it into reviewable PRs: 1. plugin skeleton, `Iceberg catalog` metadata type, VFS FileIO, type mapping 2. `Iceberg table input` 3. `Iceberg table output` with the commit handling 4. integration tests (REST catalog and MinIO in the docker harness), docs and a sample Two questions: - Is a separate `Iceberg catalog` metadata type OK, or would you rather share one catalog type with the Spark lake table transforms? Sharing would mean moving it out of the Spark plugin. - Which catalog implementations should go in the default assembly? The AWS (Glue, S3 Tables) and Hive ones are large, so I was thinking of making them optional. If this direction sounds right, I'll start with PR 1. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
