[
https://issues.apache.org/jira/browse/FLINK-7755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16210847#comment-16210847
]
ASF GitHub Bot commented on FLINK-7755:
---------------------------------------
GitHub user fhueske opened a pull request:
https://github.com/apache/flink/pull/4858
[FLINK-7755] [table] Fix NULL handling in batch joins.
## What is the purpose of the change
This PR fixes a couple of issues with Table API / SQL batch joins:
- Proper support for joining null values for inner and outer joins
- Support for non-equi join predicates in outer joins (at least one
equi-join predicate is required)
- Support for local predicates on the outer input of outer joins (at least
one equi-join predicate is required)
## Brief change log
- Inner & Outer Joins: Evaluate all join predicates in a code-gen'd
function (also equi-join predicates) for correct handling of three-value logic
- Outer joins: translate outer joins into a sequence of GroupReduce ->
OuterJoin -> GroupReduce.
- The first GroupReduce groups on the full input row and deduplicates the
outer side(s) of the join. A count for the number of deduplicated rows is kept.
- The OuterJoin evaluates the join predicate and computes possible join
pairs of left and right rows. The non-outer element of the pair can be null if
the join predicate does not match.
- The second GroupReduce groups again on the full input row and computes
for each outer row the join result. If it was not match with any inner row, it
produces a null-padded result.
- The plan for left and right outer joins requires only a single initial
partitioning and sort of each input. The all operators can reuse the initial
sort and produce a sorted result again. A full outer join requires an
additional partitioning and sorting step.
- Checks for outer join translation are removed to allow outer joins with
non-equi and local predicates.
## Verifying this change
- added ITCases for the new outer join features to `JoinITCase`
- added plan tests for Table API and SQL for the new outer join features
- updated validation tests
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): **no**
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: **no**
- The serializers: **no**
- The runtime per-record code paths (performance sensitive): **yes**
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: **no**
## Documentation
The documentation does not need to be adjusted because the outer join
limitation were not documented.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/fhueske/flink tableBatchNullJoin
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4858.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4858
----
commit 1434d8d7debe207e4b8350199eded4e678885571
Author: Fabian Hueske <[email protected]>
Date: 2017-10-15T15:55:23Z
[FLINK-7755] [table] Fix NULL handling in batch joins.
Fixes [FLINK-5498] (Add support for non-equi join and local predicates to
outer joins) as well.
----
> Null values are not correctly handled by batch inner and outer joins
> --------------------------------------------------------------------
>
> Key: FLINK-7755
> URL: https://issues.apache.org/jira/browse/FLINK-7755
> Project: Flink
> Issue Type: Bug
> Components: Table API & SQL
> Affects Versions: 1.4.0, 1.3.2
> Reporter: Fabian Hueske
> Assignee: Fabian Hueske
> Priority: Blocker
> Fix For: 1.4.0, 1.3.3
>
>
> Join predicates of batch joins are not correctly evaluated according to
> three-value logic.
> This affects inner as well as outer joins.
> The problem is that some equality predicates are only evaluated by the
> internal join algorithms of Flink which are based on {{TypeComparator}}. The
> field {{TypeComparator}} for {{Row}} are implemented such that {{null ==
> null}} results in {{TRUE}} to ensure correct ordering and grouping. However,
> three-value logic requires that {{null == null}} results to {{UNKNOWN}} (or
> null). The code generator implements this logic correctly, but for equality
> predicates, no code is generated.
> For outer joins, the problem is a bit tricker because these do not support
> code-generated predicates yet (see FLINK-5520). FLINK-5498 proposes a
> solution for this issue.
> We also need to extend several of the existing tests and add null values to
> ensure that the join logic is correctly implemented.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
