mattcasters opened a new pull request, #7489:
URL: https://github.com/apache/hop/pull/7489
## What changes were proposed in this pull request?
Add a **Sorted Schema Merge** transform (issue #7488), ported from the
hop-data-vault effort.
It combines Sorted Merge and Stream Schema Merge:
- Merges multiple **pre-sorted** input streams into one globally sorted
output
- Supports **different schemas** per input and produces a unified sparse row
layout
- Useful with Repeat Fields when combining multi-source / Data Vault
satellite data into SCD2-style streams
Also includes:
- Unit tests for schema union, sort-key resolution, and compare logic
- User manual docs and transform icon
- Integration test `main-0106-sorted-schema-merge` with a golden sparse-row
data set
## What was tested?
- `./mvnw -pl plugins/transforms/sortedschemamerge spotless:apply test` (5
unit tests)
- Ran `main-0106-sorted-schema-merge.hwf` via hop-run against a local Hop
client with the new plugin installed; unit test and golden data set validation
passed
## How can this be tested?
1. Build and install the `sortedschemamerge` transform plugin
2. Create a pipeline with 2+ pre-sorted streams of different layouts →
Sorted Schema Merge (sort key) → Dummy
3. Or run the transforms integration test:
- `main-0106-sorted-schema-merge.hwf` / unit test
`0106-sorted-schema-merge UNIT`
Fixes: #7488
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
