[
https://issues.apache.org/jira/browse/FLINK-40902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18124507#comment-18124507
]
Fabian Hueske commented on FLINK-40902:
---------------------------------------
*Root cause*
`LogicalJoinToLateralSnapshotJoinRule` required an equality predicate
(`analyzeCondition().leftKeys.isEmpty()`). This misfires when a local filter
pins the join key to a constant, e.g. `ON o.customer_id = c.customer_id WHERE
o.customer_id = 3200`: Calcite pushes the constant to both sides and simplifies
the join condition to `TRUE`, so no equi-key remains and the rule wrongly
rejects the query.
*Fix*
Lift the equi-key requirement instead of reconstructing the folded key (which
is gone before the rule runs and can't be recovered reliably). This aligns
`LATERAL SNAPSHOT` with other Flink joins that already tolerate a missing
equi-key and run single-threaded:
- *Streaming:* `SINGLETON` distribution — the operator already supports an
empty key, so no operator change.
- *Batch:* broadcast nested-loop join (build side broadcast) instead of a hash
join.
A join with an equi-key is unchanged.
*Why it's safe:* with no equi-key the whole predicate lives in the join
condition and nothing relies on key co-location, so the single-threaded plan is
correct — the same fallback other joins use. The single-threadedness is
inherent to the query (one key value), and is shown honestly as
`distribution=[single]` / a broadcast nested-loop join.
*Testing:* plan tests assert the keyless plans (streaming `SINGLETON`, batch
broadcast nested-loop); stream and batch semantic test programs cover keyless
INNER/LEFT non-equi and cross (`TRUE`) joins end-to-end on both backends; and
an operator harness test exercises the empty-key path across the LOAD and JOIN
phases. Docs updated: an equality predicate is recommended for parallelism but
no longer required.
> LATERAL SNAPSHOT join wrongly rejected when a local filter targets the join
> key column
> --------------------------------------------------------------------------------------
>
> Key: FLINK-40902
> URL: https://issues.apache.org/jira/browse/FLINK-40902
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
> Reporter: Fabian Hueske
> Assignee: Fabian Hueske
> Priority: Major
>
> A LATERAL SNAPSHOT join that has a valid equi-join predicate is wrongly
> rejected with
> "LATERAL SNAPSHOT join requires at least one equality predicate."
> whenever the query also has a local filter on the same column that carries the
> equi-join predicate.
> Example (fails):
> SELECT o.order_id, c.city
> FROM orders AS o
> JOIN LATERAL SNAPSHOT(
> input => TABLE customers,
> on_time => DESCRIPTOR(rowtime)
> ) AS c
> ON o.customer_id = c.customer_id
> WHERE o.customer_id = 3200;
> A filter on any other (non-join-key) column, e.g. WHERE o.amount = 42, works
> fine.
> Root cause:
> LogicalJoinToLateralSnapshotJoinRule enforces the equi-key requirement via
> join.analyzeCondition().leftKeys.isEmpty(). When the local filter targets the
> join key, Calcite's constant propagation pushes o.customer_id = 3200 to the
> probe
> side, infers customers.customer_id = 3200 on the build side, and then
> simplifies
> the join condition o.customer_id = c.customer_id to TRUE. analyzeCondition()
> then
> reports no equi-keys and the rule rejects a join that is in fact an
> equi-join. The
> error message is also misleading, since an equality predicate was provided.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
