mattcasters commented on issue #7077:
URL: https://github.com/apache/hop/issues/7077#issuecomment-4573482683
Desired high level architecture:
```
[DV Metadata Model (.json)] ──> [Run Request] ──> [In-Memory Compiling
(PipelineMeta)] ──> [Immediate Execution]
```
Phase 1: Establish the Data Vault Model Schema (@HopMetadata)
Phase 2: Build the Visual Modeling Perspective (Hop GUI)
Phase 3: Develop the Custom Workflow Action (@Action)
Phase 4: Debugging and Catalog Optimization
Key design principles:
Zero Deployments: The model lives as pure metadata.
Zero Database Overload: No heavy, expensive SQL joins are forced onto the
data warehouse for delta checking.
Minimal I/O: Millions of unchanged rows are dropped in memory before they
ever hit the network or storage layer.
On CDC for satelites:
```
[Source DB / Iceberg] ──> (SQL with ORDER BY BK) ──> [Hop Stream 1] ──┐
├──>
[Merge Rows (diff)] ──> [Filter: Changed/New] ──> [Target Append]
[Target Satellite] ──> (SQL with ORDER BY BK) ──> [Hop Stream 2] ───┘
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
