[
https://issues.apache.org/jira/browse/FLINK-40420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18114706#comment-18114706
]
Bettle commented on FLINK-40420:
--------------------------------
I have completed the implementation for FLINK-40420 and opened [PR
#29166|https://github.com/apache/flink/pull/29166]. All 351 local DataFrame
tests pass. The PR remains a draft pending further validation, including
community CI.
During testing, I found an existing Table API issue with mixed INT/BIGINT
inputs. It reproduces through direct Table API calls without importing or using
DataFrame, so it is not introduced by the new wrappers.
*Baseline behavior*
On the unmodified baseline, {{intersect}}, {{minus}}, {{intersect_all}}, and
{{minus_all}} resolve a BIGINT result schema but fail when executed. {{union}}
and {{union_all}} work with the same inputs. Explicitly casting the INT input
to BIGINT makes all six operations succeed.
*Comparison with existing PRs*
To check whether existing work addresses this, I applied the patches from [PR
#28533|https://github.com/apache/flink/pull/28533] (FLINK-39991) and [PR
#28522|https://github.com/apache/flink/pull/28522] (FLINK-39985) *separately*
to the same baseline ({{e01bbcacd96}}), rebuilt the Planner, and ran the same
18 direct Table API execution cases for each version.
Results for mixed INT/BIGINT inputs:
||Tested version||intersect||minus||intersect_all||minus_all||
|Baseline, no patches|FAIL|FAIL|FAIL|FAIL|
|Baseline + PR #28533|PASS|PASS|FAIL|FAIL|
|Baseline + PR #28522|FAIL|PASS|FAIL|FAIL|
PR #28533's current patch replaces *both DISTINCT rules*, not just the
intersection rule. With that patch, both {{intersect}} and {{minus}} succeed in
this reproduction. PR #28522 alone makes {{minus}} succeed.
However, *both ALL variants still fail after either patch*, reporting an input
type mismatch in {{UnionTransformation}}. The same-type and explicit-cast
control cases pass in all three versions.
*Validation scope*
These were targeted batch-mode execution tests on the 2.4-SNAPSHOT baseline
with Calcite 1.41.0, not full Planner regression runs. Apart from formatting
adjustments to the Scala test files, I made no changes to the imported patches.
*Proposed next step*
I have kept planner changes out of FLINK-40420. The DataFrame wrappers delegate
to the existing Table API, and the affected method docstrings advise using
explicit casts when input types differ. A direct Table API reproduction is
included in the PR description.
Would it make sense to open a separate bug for the remaining mixed-type
{{intersect_all}} and {{minus_all}} failures, referencing FLINK-39991 and
FLINK-39985, while keeping PR #29166 focused on the DataFrame wrappers?
I can attach the comparison tests and full error logs to support that follow-up.
> Add set operations to DataFrame API
> -----------------------------------
>
> Key: FLINK-40420
> URL: https://issues.apache.org/jira/browse/FLINK-40420
> Project: Flink
> Issue Type: Sub-task
> Components: API / Python
> Reporter: Liu Liu
> Assignee: Bettle
> Priority: Major
> Labels: pull-request-available
>
> Add relational set operations to DataFrame:
> {code:java}
> def union(self, other: DataFrame) -> DataFrame
> def union_all(self, other: DataFrame) -> DataFrame
> def intersect(self, other: DataFrame) -> DataFrame
> def intersect_all(self, other: DataFrame) -> DataFrame
> def minus(self, other: DataFrame) -> DataFrame
> def minus_all(self, other: DataFrame) -> DataFrame {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
