Tim Armstrong created IMPALA-9660:
-------------------------------------
Summary: Distributed codegen
Key: IMPALA-9660
URL: https://issues.apache.org/jira/browse/IMPALA-9660
Project: IMPALA
Issue Type: Improvement
Components: Distributed Exec
Reporter: Tim Armstrong
Another potential extension of IMPALA-5444 is that we can distribute the
codegen work of different fragments across different backends. Today, each
fragment will generate the same code on each backend server it's assigned to
run on. This is mostly redundant work (except for scan nodes if different scan
ranges correspond to different file formats). It would be great to consolidate
the code generation work items among the backend servers and avoids redundant
work. The codegen for a fragment (or an exec node if we allow ourselves to use
multiple LLVM modules per fragment so as to allow parallel codegen for
different exec nodes in a fragment) could be assigned to backend servers and
the compiled code can be shipped to the backend Impalad servers when it's
ready. Of course, this may involve some security issues as we have to trust the
binary being shipped over. We may also need to take into account of the latency
for shipping the code. However, this is potentially a huge saving in CPUs for
queries with many fragments running on a huge cluster.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
