bamaer commented on PR #8807:
URL: https://github.com/apache/hop/pull/8807#issuecomment-6066930323
Docs-only, so none of this can break anything. I reviewed the text but
didn't build the site.
**Fix before merge**
1. **Should `from_ts` be the row's own `changed_date_ts` rather than Lag
1?** (`high-volume-scd2.adoc`, "Calculate the validity range"). With Lag, each
version starts when the version before it started, so neighbouring ranges
overlap, as the page itself says. That causes two problems further down the
guide:
- The fact join "range contains the fact timestamp" matches two SCD2 rows
for any fact older than the latest change.
- On an incremental load the stored current row is the first row of its
key, so its recomputed `from_ts` becomes `1900-01-01`. The Update on
`business_key` + `from_ts` then misses that row, or closes the key's first
version instead.
Using `from_ts = changed_date_ts` and `to_ts` = Lead 1 gives ranges that
don't overlap and fixes both. The example tables and the Analytic Query
screenshot would need to change with it.
2. **An incremental load inserts unchanged current rows again**
("Incremental load"). The stored current row is fed back in as a source. For a
key with no new change, or only changes that the Group By collapses into it,
that row comes out with `to_ts = 9999-12-31`, and "Bulk load every row that
still ends at `9999-12-31`" inserts a duplicate. Suggested fix: only insert
rows that don't come from the target source. Filter on `record_source`, and
only read target rows for business keys that appear in the delta.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
