fresh-borzoni commented on PR #27372: URL: https://github.com/apache/flink/pull/27372#issuecomment-4359995004
Real-world case demonstrating the need.
Flink 2.2.0 + Fluss 0.9, classic parent/child CDC join - sale_items.PK =
(provider_id, sale_id, id) strictly extends sales.PK = (provider_id, id), with
bucket key on each side covering its join key. All documented DeltaJoin
preconditions hold (Fluss PK on both sides, INNER JOIN, no DELETE,
delta-join.strategy=AUTO).
``` sql
INSERT INTO sales_with_sale_items
SELECT s.provider_id, s.id AS sale_id, si.id AS sale_item_id, ...
FROM sale_items si INNER JOIN sales s
ON si.provider_id = s.provider_id AND si.sale_id = s.id;
```
Stock 2.2.0 falls back to a regular Join and injects a
SinkUpsertMaterializer:
+- Sink(... upsertMaterialize=[true])
+- Calc(...)
+- Join(leftInputSpec=[HasUniqueKey],
rightInputSpec=[JoinKeyContainsUniqueKey],
changelogMode=[I,UB,UA], duplicateChanges=[DISALLOW])
:- TableSourceScan([sale_items])
+- TableSourceScan([sales])
This PR cherry-picked onto release-2.2.0:
+- Sink(...)
+- DeltaJoin(...)
:- DropUpdateBefore +- TableSourceScan([sale_items])
+- DropUpdateBefore +- TableSourceScan([sales])
Why: Base getJoinUpsertKeys returns only {si.provider_id, si.sale_id, si.id}
(left's PK). The natural SELECT projects s.provider_id, s.id - bit-identical
under INNER JOIN, but the Calc only retains projected columns, so the only
upsert key dies. No upstream key -> SinkUpsertMaterializer ->
duplicateChanges=DISALLOW propagates -> DropUpdateBefore blocked → DeltaJoin
rejected.
This PR's enrichment derives {si.id, s.provider_id, s.id} via the equi-pairs
- survives the Calc, matches the sink PK, materializer drops, DeltaJoin
applies.
The asymmetric parent/child shape (one PK strictly extends the join key, the
other matches it) is the canonical CDC join - orders/order_items,
users/sessions, conversations/messages, etc. A SELECT-rewrite workaround exists
(project FK columns from the child instead of the parent), but the failure mode
is silent and the gap isn't obvious to authors.
@lincoln-lil Can you take a look, pls?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
