Dandandan commented on PR #5837:
URL:
https://github.com/apache/arrow-datafusion/pull/5837#issuecomment-1494393563
> > As far as I can see, this only works for single partitions as input and
not repartitioning in between (e.g. no concurrency), could you confirm?
>
> No always. We will see the adjacent Partial + Final Aggregator for normal
join and aggregation on the same key. I will add more UTs and intg tests
tomorrow to show the cases:
>
> ```rust
> select distinct(t1.t1_id) from t1 inner join t2 on t1.t1_id = t2.t2_id;
> ```
>
> ```sql
> AggregateExec: mode=Single, gby=[t1_id@0 as t1_id], aggr=[]",
> ProjectionExec: expr=[t1_id@0 as t1_id]",
> CoalesceBatchesExec: target_batch_size=4096",
> HashJoinExec: mode=Partitioned, join_type=Inner, on=[(Column {
name: \"t1_id\", index: 0 }, Column { name: \"t2_id\", index: 0 })]",
> CoalesceBatchesExec: target_batch_size=4096",
> RepartitionExec: partitioning=Hash([Column { name:
\"t1_id\", index: 0 }], 2), input_partitions=2",
> RepartitionExec: partitioning=RoundRobinBatch(2),
input_partitions=1",
> MemoryExec: partitions=1, partition_sizes=[1]",
> CoalesceBatchesExec: target_batch_size=4096",
> RepartitionExec: partitioning=Hash([Column { name:
\"t2_id\", index: 0 }], 2), input_partitions=2",
> RepartitionExec: partitioning=RoundRobinBatch(2),
input_partitions=1",
> MemoryExec: partitions=1, partition_sizes=[1]",
> ```
Ah yes - in the case the underlying partition is already hash-repartitioned
on the key. Makes sense, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
