Dandandan commented on issue #3629:
URL:
https://github.com/apache/arrow-datafusion/issues/3629#issuecomment-1260004551
Yeah so the current optimization pass will add the `RepartitionExec`
whenever the `target_partitions` is bigger than the incoming partitions - that
definitely is something to improve upon. Plus your observation that no
`RepartitionExec` is added when you don't add a `repartition`.
I would argue a more sensical final result should look like this (regardless
of adding a `repartition` but based on your cores or configured
`target_partitions` ):
```
ParquetExec ...
RepartitionExec: partitioning=RoundRobinBatch(20)
CsvExec ...
```
Because no part of the query *requires* to have exactly 4 partitions
available, so the optimizer is free to increase / descrease it where it sees
fit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
