hanyuzheng7 opened a new pull request, #29724:
URL: https://github.com/apache/flink/pull/29724
## What is the purpose of the change
With `table.optimizer.join-reorder-enabled = true`,
`FlinkBushyJoinReorderRule` drops join conditions, so queries return wrong
results. The bushy rule reorders every join with at most
`table.optimizer.bushy-join-reorder-threshold` (12 by default) inputs, in batch
and in streaming mode.
For example, this query returns all 1000 rows instead of 1, because the
WHERE filter is dropped:
```sql
WITH d AS (SELECT n FROM (VALUES (0),(1),(2),(3),(4),(5),(6),(7),(8),(9)) AS
t(n))
SELECT d1.n + 10 * d2.n + 100 * d3.n, * FROM d d1, d d2, d d3
WHERE d1.n + 10 * d2.n + 100 * d3.n = 1;
```
## Brief change log
The rule only applied the join conditions that reference both sides of a
join it built and no other factor. It dropped the others:
- a join filter that references three factors of a cross join (the query
above), or the null generating factor of an outer join and another factor, e.g.
`b2 > b3` in `T1 LEFT JOIN T2 ON a1 = a2 JOIN T3 ON a1 = a3 AND b2 > b3`,
- a conjunct of an outer join condition that references only one side, e.g.
`b1 > 10` in `T1 LEFT JOIN T2 ON a1 = a2 AND b1 > 10`,
- an outer join condition that references a factor that wasn't joined yet,
e.g. the null generating factor of another outer join. The outer join got the
condition `true`.
It also joined a full outer join without its condition as an inner join (`T1
FULL JOIN T2 ON b1 > 10`), and kept the wrong side of a right outer join whose
condition is `true` (`T1 RIGHT JOIN T2 ON b1 > 10`, after `b1 > 10` is pushed
into T1).
Changes in `FlinkBushyJoinReorderRule`:
- Each outer join is joined as a left outer join with its whole condition,
after the factors that the condition references.
- The other factors are joined with the join filters they can apply.
- The join filters that no join applies are applied on top of the join tree,
with the post-join filter.
## Verifying this change
- Added 7 cases to `JoinReorderTestBase` (batch and stream plans) and 7 to
`JoinReorderITCaseBase` (batch and streaming), for the cases above. Without the
fix, all of them fail with the bushy reorder. The 4-table case also runs with
LOPT, which doesn't have the bug.
- The expected plan of the existing
`testLeftOuterJoinInnerJoinLeftOuterJoinInnerJoin[isBushyJoinReorder=true]` had
the bug: it dropped `a4 = a5` and joined T5 with `where=[true]`. It now joins
T5 with `where=[=(a4, a5)]`.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no. Only the plans of
queries with this bug change.
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
