[ https://issues.apache.org/jira/browse/FLINK-7755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16210847#comment-16210847 ]
ASF GitHub Bot commented on FLINK-7755: --------------------------------------- GitHub user fhueske opened a pull request: https://github.com/apache/flink/pull/4858 [FLINK-7755] [table] Fix NULL handling in batch joins. ## What is the purpose of the change This PR fixes a couple of issues with Table API / SQL batch joins: - Proper support for joining null values for inner and outer joins - Support for non-equi join predicates in outer joins (at least one equi-join predicate is required) - Support for local predicates on the outer input of outer joins (at least one equi-join predicate is required) ## Brief change log - Inner & Outer Joins: Evaluate all join predicates in a code-gen'd function (also equi-join predicates) for correct handling of three-value logic - Outer joins: translate outer joins into a sequence of GroupReduce -> OuterJoin -> GroupReduce. - The first GroupReduce groups on the full input row and deduplicates the outer side(s) of the join. A count for the number of deduplicated rows is kept. - The OuterJoin evaluates the join predicate and computes possible join pairs of left and right rows. The non-outer element of the pair can be null if the join predicate does not match. - The second GroupReduce groups again on the full input row and computes for each outer row the join result. If it was not match with any inner row, it produces a null-padded result. - The plan for left and right outer joins requires only a single initial partitioning and sort of each input. The all operators can reuse the initial sort and produce a sorted result again. A full outer join requires an additional partitioning and sorting step. - Checks for outer join translation are removed to allow outer joins with non-equi and local predicates. ## Verifying this change - added ITCases for the new outer join features to `JoinITCase` - added plan tests for Table API and SQL for the new outer join features - updated validation tests ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): **no** - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: **no** - The serializers: **no** - The runtime per-record code paths (performance sensitive): **yes** - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: **no** ## Documentation The documentation does not need to be adjusted because the outer join limitation were not documented. You can merge this pull request into a Git repository by running: $ git pull https://github.com/fhueske/flink tableBatchNullJoin Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/4858.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4858 ---- commit 1434d8d7debe207e4b8350199eded4e678885571 Author: Fabian Hueske <fhue...@apache.org> Date: 2017-10-15T15:55:23Z [FLINK-7755] [table] Fix NULL handling in batch joins. Fixes [FLINK-5498] (Add support for non-equi join and local predicates to outer joins) as well. ---- > Null values are not correctly handled by batch inner and outer joins > -------------------------------------------------------------------- > > Key: FLINK-7755 > URL: https://issues.apache.org/jira/browse/FLINK-7755 > Project: Flink > Issue Type: Bug > Components: Table API & SQL > Affects Versions: 1.4.0, 1.3.2 > Reporter: Fabian Hueske > Assignee: Fabian Hueske > Priority: Blocker > Fix For: 1.4.0, 1.3.3 > > > Join predicates of batch joins are not correctly evaluated according to > three-value logic. > This affects inner as well as outer joins. > The problem is that some equality predicates are only evaluated by the > internal join algorithms of Flink which are based on {{TypeComparator}}. The > field {{TypeComparator}} for {{Row}} are implemented such that {{null == > null}} results in {{TRUE}} to ensure correct ordering and grouping. However, > three-value logic requires that {{null == null}} results to {{UNKNOWN}} (or > null). The code generator implements this logic correctly, but for equality > predicates, no code is generated. > For outer joins, the problem is a bit tricker because these do not support > code-generated predicates yet (see FLINK-5520). FLINK-5498 proposes a > solution for this issue. > We also need to extend several of the existing tests and add null values to > ensure that the join logic is correctly implemented. -- This message was sent by Atlassian JIRA (v6.4.14#64029)