kotwal-itpro commented on issue #5524:
URL: https://github.com/apache/hop/issues/5524#issuecomment-6042686684
That makes a lot of sense, @hansva. One set of transforms that runs on both
engines is much better than two parallel ones. I had a look at how the Spark
lake table transforms are wired, and I think it can be done without breaking
existing pipelines:
**Proposed layout**
- A new plugin, `plugins/tech/lakehouse`, that owns the shared parts:
- the transform metas and dialogs for Lake table input / output / merge /
maintenance, moved out of the Spark plugin,
- the catalog metadata type (today's `Spark Catalog`), as a general
lakehouse catalog.
- The Spark plugin keeps everything Spark-specific (the
`SparkLakeTable*Handler` classes, the SQL builders, `SparkCatalogApplier`) and
reaches the moved metas through `dependencies.xml`, the same way it already
does for memgroupby, mergejoin, sort and the other transforms it supports.
- The local engine gets real implementations for input and output, built on
the Iceberg Java library: storage through Hop VFS and one atomic snapshot per
pipeline run, as described above. Merge and maintenance stay Spark-only at
first.
**Compatibility**
- Transform plugin IDs stay the same (`SparkLakeTableInput`, ...), and the
catalog keeps `spark-catalog` as a legacy key, so existing pipelines and
metadata open unchanged. Only the display names lose the "Spark" prefix.
- On the local engine, Delta tables give a clear "not supported on this
engine yet" message, until Delta Kernel support lands (#3941).
**PR plan**
1. Pure move: the new lakehouse plugin, metas, dialogs and catalog type move
over, and the Spark plugin depends on them. No behavior change, and the
existing Spark Iceberg/Delta integration tests should stay green.
2. Local engine read support (Iceberg).
3. Local engine write support (Iceberg).
4. Integration tests for the local engine, docs and samples.
Keeping the move separate from the new behavior should make both much easier
to review.
@mattcasters since this moves code you wrote recently in #7539, are you OK
with this direction, or would you prefer the shared pieces split differently?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
