[
https://issues.apache.org/jira/browse/IMPALA-9660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Csaba Ringhofer reassigned IMPALA-9660:
---------------------------------------
Assignee: Daniel Becker
> Distributed codegen
> -------------------
>
> Key: IMPALA-9660
> URL: https://issues.apache.org/jira/browse/IMPALA-9660
> Project: IMPALA
> Issue Type: Improvement
> Components: Distributed Exec
> Reporter: Tim Armstrong
> Assignee: Daniel Becker
> Priority: Major
> Labels: codegen
>
> Another potential extension of IMPALA-5444 is that we can distribute the
> codegen work of different fragments across different backends. Today, each
> fragment will generate the same code on each backend server it's assigned to
> run on. This is mostly redundant work (except for scan nodes if different
> scan ranges correspond to different file formats). It would be great to
> consolidate the code generation work items among the backend servers and
> avoids redundant work. The codegen for a fragment (or an exec node if we
> allow ourselves to use multiple LLVM modules per fragment so as to allow
> parallel codegen for different exec nodes in a fragment) could be assigned to
> backend servers and the compiled code can be shipped to the backend Impalad
> servers when it's ready. Of course, this may involve some security issues as
> we have to trust the binary being shipped over. We may also need to take into
> account of the latency for shipping the code. However, this is potentially a
> huge saving in CPUs for queries with many fragments running on a huge cluster.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
