andygrove opened a new issue, #2573:
URL: https://github.com/apache/arrow-datafusion/issues/2573
**Is your feature request related to a problem or challenge? Please describe
what you are trying to do.**
The current support for `UNION` vs `UNION ALL` is confusing to me.
- The logical plan simply has the `Union` operator and this is currently
assumed to represent `UNION ALL`.
- The SQL planner has logic to wrap a union with an aggregate query and that
seems like something we would want in the physical plan but not in the logical
plan
- The DataFrame API does not have a function for performing a regular
`UNION`. It is possible to manually wrap a union in a distinct to achieve this
but that might not be obvious to users. Also, the documentation for `distinct`
is incorrect and says that it performs a `union`
Here is our logical plan for a UNION query:
```
❯ explain select * from foo union select * from foo;
+---------------+----------------------------------------------------------------------------------------------------------------+
| plan_type | plan
|
+---------------+----------------------------------------------------------------------------------------------------------------+
| logical_plan | Projection: #a, #b
|
| | Aggregate: groupBy=[[#a, #b]], aggr=[[]]
|
| | Union
|
| | Projection: #foo.a, #foo.b
|
| | TableScan: foo projection=Some([0, 1])
|
| | Projection: #foo.a, #foo.b
|
| | TableScan: foo projection=Some([0, 1])
|
+---------------+----------------------------------------------------------------------------------------------------------------+
```
**Describe the solution you'd like**
I think what we want is:
- Logical plan should represent `UNION` vs `UNION ALL`
- The DataFrame API should have functions for both `union` (existing method
representing `UNION ALL`) and `union_distinct`
- The physical planner (or maybe an optimization rule) should translate a
`UNION` to an aggregate query
**Describe alternatives you've considered**
Leave things as they are and improve the documentation.
**Additional context**
For users of DataFusion for SQL query planning, it would be easier to map
union/union all to other engines rather than trying to reverse engineer the
aggregate query wrapping the union.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
