Siddharth Seth created TEZ-2003:
-----------------------------------
Summary: [Umbrella] Allow Tez to co-ordinate execution to external
services
Key: TEZ-2003
URL: https://issues.apache.org/jira/browse/TEZ-2003
Project: Apache Tez
Issue Type: Improvement
Reporter: Siddharth Seth
The Tez engine itself takes care of co-ordinating execution - controlling how
data gets routed (different connection patterns), fault tolerance, scheduling
of work, etc.
This is currently tied to TaskSpecs defined within Tez and on containers
launched by Tez itself (TezChild).
The proposal is to allow Tez to work with external services instead of just
containers launched by Tez. This involves several more pluggable layers to work
with alternate Task Specifications, custom launch and task allocation
mechanics, as well as custom scheduling sources.
A simple example would be a simple a process with the capability to execute
multiple Tez TaskSpecs as threads. In such a case, a container launch isn't
really need and can be mocked. Sourcing / scheduling containers would need to
be pluggable.
A more advanced example would be LLAP (HIVE-7926;
https://issues.apache.org/jira/secure/attachment/12665704/LLAPdesigndocument.pdf).
This works with custom interfaces - which would need to be supported by Tez,
along with a custom event model which would need translation hooks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
