wirybeaver opened a new pull request, #26875:
URL: https://github.com/apache/datafusion/pull/26875
## Which issue does this PR close?
DataFusion prerequisite for apache/datafusion-ballista#1677 (D0); this PR
does not close the roadmap.
## Rationale for this change
A provider's external key grouping can avoid a join shuffle, but equal
partition counts do not prove that two inputs use the same groups. Labelling an
external bucket function as native Hash can also enable incorrect partition
routing.
## What changes are included in this PR?
Propose `Partitioning::Grouped`: keys, a versioned semantic scheme, and
ordered group identities. Existing distribution requirements establish
compatibility; projection and joins preserve or discard the guarantee as
appropriate. The property is serialized, while FFI conservatively exposes
Unknown partitioning.
```text
provider grouping -> Grouped(keys, scheme, IDs)
|
distribution requirements
/ \
compatible incompatible
| |
retain grouping repartition
```
This draft isolates the property contract for upstream review. Ballista
currently integrates a separate 55.2 backport; this branch is based on
DataFusion main.
## Are these changes tested?
The Ballista integration executes real bucketed Parquet through task
serialization and compares results with ordinary shuffle joins, including
missing groups, duplicate/null keys, projection and downstream native joins.
Those end-to-end results belong to the 55.2 integration branch, not this main
port.
## Are there any user-facing changes?
Adds a public Partitioning variant and its physical protobuf representation.
Providers must supply complete grouping semantics; no native hash router is
implied.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
