[
https://issues.apache.org/jira/browse/ARROW-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16949779#comment-16949779
]
Kyle McCarthy edited comment on ARROW-6659 at 10/11/19 8:32 PM:
----------------------------------------------------------------
I am happy to help - and I would prefer to do it how you wanted/the right way!
I am fairly unfamiliar with the codebase so I am really just learning it by
working through the open tasks, so this may be a dumb question.
How does the LogicalPlan and partition count actually work together. From the
tests it looks like the partition count is related to the batch size? If so
that would mean that every LogicalPlan would have the same partition count
right?
Also, if we do add the LogicalPlan::Merge - that would mean that when the SQL
Planner is creating a Logical Aggregate it would create:
{code:java}
Aggregate { Merge { Aggregate ( aggregate_input ) } }{code}
? If so that definitely makes sense to me, but I am still not totally sure how
the partition count would work into this.
Thank you for your patience!
was (Author: kylemccarthy):
I am happy to help - and I would prefer to do it how you wanted/the right way!
I am fairly unfamiliar with the codebase so I am really just learning it by
working through the open tasks, so this may be a dumb question.
How does the LogicalPlan and partition count actually work together. From the
tests it looks like the partition count is related to the batch size? If so
that would mean that every LogicalPlan would have the same partition count
right?
Also, if we do add the LogicalPlan::Merge - that would mean that when the SQL
Planner is creating a Logical Aggregate it would create: `Aggregate { Merge {
Aggregate ( aggregate_input ) } }`? If so that definitely makes sense to me,
but I am still not totally sure how the partition count would work into this.
Thank you for your patience!
> [Rust] [DataFusion] Refactor of HashAggregateExec to support custom merge
> -------------------------------------------------------------------------
>
> Key: ARROW-6659
> URL: https://issues.apache.org/jira/browse/ARROW-6659
> Project: Apache Arrow
> Issue Type: Sub-task
> Components: Rust, Rust - DataFusion
> Reporter: Andy Grove
> Assignee: Kyle McCarthy
> Priority: Major
>
> HashAggregateExec current creates one HashPartition per input partition for
> the initial aggregate per partition, and then explicitly calls MergeExec and
> then creates another HashPartition for the final reduce operation.
> This is fine for in-memory queries in DataFusion but is not extensible. For
> example, it is not possible to provide a different MergeExec implementation
> that would distribute queries to a cluster.
> A better design would be to move the logic into the query planner so that the
> physical plan contains explicit steps such as:
>
> {code:java}
> - HashAggregate // final aggregate
> - MergeExec
> - HashAggregate // aggregate per partition
> {code}
> This would then make it easier to customize the plan in other projects, to
> support distributed execution:
> {code:java}
> - HashAggregate // final aggregate
> - MergeExec
> - DistributedExec
> - HashAggregate // aggregate per partition{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
