mustafasrepo commented on issue #9370:
URL:
https://github.com/apache/arrow-datafusion/issues/9370#issuecomment-1969028599
I think, combining consecutive
```
RepartitionExec(hash)
--RepartitionExec(round robin)
```
into
```
RepartitionExec(hash) (where inputs hashed in parallel)
```
produces much more readable plans. And I presume it would be better in terms
of execution speed. Hence we should have this support. However, combining
`CoalesceBatchesExec` into `RepartitionExec` may introduce complications as
suggested by @ozankabak . I think, one of the causes of this problem is that
`CoalesceBatches` rule is to naive in its current state.
By refactoring `CoalesceBatches` rule, we can produce better plans as is.
As an example, `CoalesceBatchesExec` in the build side of the `HashJoinExec`
is unnecesary (given that `HashJoinExec` already buffers up all data at its
input). However, this kind of analysis may require new `API`s. We can also
benefit from estimated selectivity in the `Statistics` for the
`CoalesceBatchesExec` insertion decision.
```
ProjectionExec: expr=[name@1 as schoolname, name@3 as teachername]
CoalesceBatchesExec: target_batch_size=8192
HashJoinExec: mode=Partitioned, join_type=Inner, on=[(id@0, class_id@0)]
CoalesceBatchesExec: target_batch_size=8192
RepartitionExec: partitioning=Hash([id@0], 8), input_partitions=8
RepartitionExec: partitioning=RoundRobinBatch(8),
input_partitions=1
VirtualExecutionPlan
CoalesceBatchesExec: target_batch_size=8192
RepartitionExec: partitioning=Hash([class_id@0], 8),
input_partitions=8
RepartitionExec: partitioning=RoundRobinBatch(8),
input_partitions=1
ProjectionExec: expr=[class_id@1 as class_id, name@2 as name]
VirtualExecutionPlan
```
Hence, I propose as a first step we should first do 2nd step in the @alamb s
suggestion. In the meantime, I will explore how can we refactor
`CoalesceBatches` rule for better plans. I presume these 2 steps together would
be sufficient for most of the use cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
