Fabian Hueske created FLINK-5722:
------------------------------------
Summary: Implement DISTINCT as dedicated operator
Key: FLINK-5722
URL: https://issues.apache.org/jira/browse/FLINK-5722
Project: Flink
Issue Type: Improvement
Components: Table API & SQL
Affects Versions: 1.2.0, 1.3.0
Reporter: Fabian Hueske
DISTINCT is currently implemented for batch Table API / SQL as an aggregate
which groups on all fields. Grouped aggregates are implemented as GroupReduce
with sort-based combiner.
This operator can be more efficiently implemented by using ReduceFunction and
hinting a HashCombine strategy. The same ReduceFunction can be used for all
DISTINCT operations and can be assigned with appropriate forward field
annotations.
We would need a custom conversion rule which translates distinct aggregations
(grouping on all fields and returning all fields) into a custom DataSetRelNode.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
