GitHub user fhueske opened a pull request:
https://github.com/apache/flink/pull/4858
[FLINK-7755] [table] Fix NULL handling in batch joins.
## What is the purpose of the change
This PR fixes a couple of issues with Table API / SQL batch joins:
- Proper support for joining null values for inner and outer joins
- Support for non-equi join predicates in outer joins (at least one
equi-join predicate is required)
- Support for local predicates on the outer input of outer joins (at least
one equi-join predicate is required)
## Brief change log
- Inner & Outer Joins: Evaluate all join predicates in a code-gen'd
function (also equi-join predicates) for correct handling of three-value logic
- Outer joins: translate outer joins into a sequence of GroupReduce ->
OuterJoin -> GroupReduce.
- The first GroupReduce groups on the full input row and deduplicates the
outer side(s) of the join. A count for the number of deduplicated rows is kept.
- The OuterJoin evaluates the join predicate and computes possible join
pairs of left and right rows. The non-outer element of the pair can be null if
the join predicate does not match.
- The second GroupReduce groups again on the full input row and computes
for each outer row the join result. If it was not match with any inner row, it
produces a null-padded result.
- The plan for left and right outer joins requires only a single initial
partitioning and sort of each input. The all operators can reuse the initial
sort and produce a sorted result again. A full outer join requires an
additional partitioning and sorting step.
- Checks for outer join translation are removed to allow outer joins with
non-equi and local predicates.
## Verifying this change
- added ITCases for the new outer join features to `JoinITCase`
- added plan tests for Table API and SQL for the new outer join features
- updated validation tests
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): **no**
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: **no**
- The serializers: **no**
- The runtime per-record code paths (performance sensitive): **yes**
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: **no**
## Documentation
The documentation does not need to be adjusted because the outer join
limitation were not documented.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/fhueske/flink tableBatchNullJoin
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4858.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4858
----
commit 1434d8d7debe207e4b8350199eded4e678885571
Author: Fabian Hueske <[email protected]>
Date: 2017-10-15T15:55:23Z
[FLINK-7755] [table] Fix NULL handling in batch joins.
Fixes [FLINK-5498] (Add support for non-equi join and local predicates to
outer joins) as well.
----
---
