[jira] [Updated] (TEZ-2003) [Umbrella] Allow Tez to co-ordinate execution to external services
[ https://issues.apache.org/jira/browse/TEZ-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated TEZ-2003: Attachment: 2003_20150821.1.txt Hopefully the final diff before merge. [Umbrella] Allow Tez to co-ordinate execution to external services -- Key: TEZ-2003 URL: https://issues.apache.org/jira/browse/TEZ-2003 Project: Apache Tez Issue Type: Improvement Reporter: Siddharth Seth Attachments: 2003_20150728.1.txt, 2003_20150807.1.txt, 2003_20150807.2.txt, 2003_20150812.1.txt, 2003_20150812.2.txt, 2003_20150814.1.txt, 2003_20150814.2.txt, 2003_20150820.1.txt, 2003_20150821.1.txt, Tez With External Services.pdf The Tez engine itself takes care of co-ordinating execution - controlling how data gets routed (different connection patterns), fault tolerance, scheduling of work, etc. This is currently tied to TaskSpecs defined within Tez and on containers launched by Tez itself (TezChild). The proposal is to allow Tez to work with external services instead of just containers launched by Tez. This involves several more pluggable layers to work with alternate Task Specifications, custom launch and task allocation mechanics, as well as custom scheduling sources. A simple example would be a simple a process with the capability to execute multiple Tez TaskSpecs as threads. In such a case, a container launch isn't really need and can be mocked. Sourcing / scheduling containers would need to be pluggable. A more advanced example would be LLAP (HIVE-7926; https://issues.apache.org/jira/secure/attachment/12665704/LLAPdesigndocument.pdf). This works with custom interfaces - which would need to be supported by Tez, along with a custom event model which would need translation hooks. Tez should be able to work with a combination of certain vertices running in external services and others running in regular Tez containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2003) [Umbrella] Allow Tez to co-ordinate execution to external services
[ https://issues.apache.org/jira/browse/TEZ-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated TEZ-2003: Attachment: 2003_20150820.1.txt [Umbrella] Allow Tez to co-ordinate execution to external services -- Key: TEZ-2003 URL: https://issues.apache.org/jira/browse/TEZ-2003 Project: Apache Tez Issue Type: Improvement Reporter: Siddharth Seth Attachments: 2003_20150728.1.txt, 2003_20150807.1.txt, 2003_20150807.2.txt, 2003_20150812.1.txt, 2003_20150812.2.txt, 2003_20150814.1.txt, 2003_20150814.2.txt, 2003_20150820.1.txt, Tez With External Services.pdf The Tez engine itself takes care of co-ordinating execution - controlling how data gets routed (different connection patterns), fault tolerance, scheduling of work, etc. This is currently tied to TaskSpecs defined within Tez and on containers launched by Tez itself (TezChild). The proposal is to allow Tez to work with external services instead of just containers launched by Tez. This involves several more pluggable layers to work with alternate Task Specifications, custom launch and task allocation mechanics, as well as custom scheduling sources. A simple example would be a simple a process with the capability to execute multiple Tez TaskSpecs as threads. In such a case, a container launch isn't really need and can be mocked. Sourcing / scheduling containers would need to be pluggable. A more advanced example would be LLAP (HIVE-7926; https://issues.apache.org/jira/secure/attachment/12665704/LLAPdesigndocument.pdf). This works with custom interfaces - which would need to be supported by Tez, along with a custom event model which would need translation hooks. Tez should be able to work with a combination of certain vertices running in external services and others running in regular Tez containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2003) [Umbrella] Allow Tez to co-ordinate execution to external services
[ https://issues.apache.org/jira/browse/TEZ-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated TEZ-2003: Attachment: 2003_20150814.2.txt [Umbrella] Allow Tez to co-ordinate execution to external services -- Key: TEZ-2003 URL: https://issues.apache.org/jira/browse/TEZ-2003 Project: Apache Tez Issue Type: Improvement Reporter: Siddharth Seth Attachments: 2003_20150728.1.txt, 2003_20150807.1.txt, 2003_20150807.2.txt, 2003_20150812.1.txt, 2003_20150812.2.txt, 2003_20150814.1.txt, 2003_20150814.2.txt, Tez With External Services.pdf The Tez engine itself takes care of co-ordinating execution - controlling how data gets routed (different connection patterns), fault tolerance, scheduling of work, etc. This is currently tied to TaskSpecs defined within Tez and on containers launched by Tez itself (TezChild). The proposal is to allow Tez to work with external services instead of just containers launched by Tez. This involves several more pluggable layers to work with alternate Task Specifications, custom launch and task allocation mechanics, as well as custom scheduling sources. A simple example would be a simple a process with the capability to execute multiple Tez TaskSpecs as threads. In such a case, a container launch isn't really need and can be mocked. Sourcing / scheduling containers would need to be pluggable. A more advanced example would be LLAP (HIVE-7926; https://issues.apache.org/jira/secure/attachment/12665704/LLAPdesigndocument.pdf). This works with custom interfaces - which would need to be supported by Tez, along with a custom event model which would need translation hooks. Tez should be able to work with a combination of certain vertices running in external services and others running in regular Tez containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2003) [Umbrella] Allow Tez to co-ordinate execution to external services
[ https://issues.apache.org/jira/browse/TEZ-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated TEZ-2003: Attachment: 2003_20150812.2.txt [Umbrella] Allow Tez to co-ordinate execution to external services -- Key: TEZ-2003 URL: https://issues.apache.org/jira/browse/TEZ-2003 Project: Apache Tez Issue Type: Improvement Reporter: Siddharth Seth Attachments: 2003_20150728.1.txt, 2003_20150807.1.txt, 2003_20150807.2.txt, 2003_20150812.1.txt, 2003_20150812.2.txt, Tez With External Services.pdf The Tez engine itself takes care of co-ordinating execution - controlling how data gets routed (different connection patterns), fault tolerance, scheduling of work, etc. This is currently tied to TaskSpecs defined within Tez and on containers launched by Tez itself (TezChild). The proposal is to allow Tez to work with external services instead of just containers launched by Tez. This involves several more pluggable layers to work with alternate Task Specifications, custom launch and task allocation mechanics, as well as custom scheduling sources. A simple example would be a simple a process with the capability to execute multiple Tez TaskSpecs as threads. In such a case, a container launch isn't really need and can be mocked. Sourcing / scheduling containers would need to be pluggable. A more advanced example would be LLAP (HIVE-7926; https://issues.apache.org/jira/secure/attachment/12665704/LLAPdesigndocument.pdf). This works with custom interfaces - which would need to be supported by Tez, along with a custom event model which would need translation hooks. Tez should be able to work with a combination of certain vertices running in external services and others running in regular Tez containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2003) [Umbrella] Allow Tez to co-ordinate execution to external services
[ https://issues.apache.org/jira/browse/TEZ-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated TEZ-2003: Attachment: 2003_20150812.1.txt [Umbrella] Allow Tez to co-ordinate execution to external services -- Key: TEZ-2003 URL: https://issues.apache.org/jira/browse/TEZ-2003 Project: Apache Tez Issue Type: Improvement Reporter: Siddharth Seth Attachments: 2003_20150728.1.txt, 2003_20150807.1.txt, 2003_20150807.2.txt, 2003_20150812.1.txt, Tez With External Services.pdf The Tez engine itself takes care of co-ordinating execution - controlling how data gets routed (different connection patterns), fault tolerance, scheduling of work, etc. This is currently tied to TaskSpecs defined within Tez and on containers launched by Tez itself (TezChild). The proposal is to allow Tez to work with external services instead of just containers launched by Tez. This involves several more pluggable layers to work with alternate Task Specifications, custom launch and task allocation mechanics, as well as custom scheduling sources. A simple example would be a simple a process with the capability to execute multiple Tez TaskSpecs as threads. In such a case, a container launch isn't really need and can be mocked. Sourcing / scheduling containers would need to be pluggable. A more advanced example would be LLAP (HIVE-7926; https://issues.apache.org/jira/secure/attachment/12665704/LLAPdesigndocument.pdf). This works with custom interfaces - which would need to be supported by Tez, along with a custom event model which would need translation hooks. Tez should be able to work with a combination of certain vertices running in external services and others running in regular Tez containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2003) [Umbrella] Allow Tez to co-ordinate execution to external services
[ https://issues.apache.org/jira/browse/TEZ-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated TEZ-2003: Attachment: 2003_20150807.2.txt [Umbrella] Allow Tez to co-ordinate execution to external services -- Key: TEZ-2003 URL: https://issues.apache.org/jira/browse/TEZ-2003 Project: Apache Tez Issue Type: Improvement Reporter: Siddharth Seth Attachments: 2003_20150728.1.txt, 2003_20150807.1.txt, 2003_20150807.2.txt, Tez With External Services.pdf The Tez engine itself takes care of co-ordinating execution - controlling how data gets routed (different connection patterns), fault tolerance, scheduling of work, etc. This is currently tied to TaskSpecs defined within Tez and on containers launched by Tez itself (TezChild). The proposal is to allow Tez to work with external services instead of just containers launched by Tez. This involves several more pluggable layers to work with alternate Task Specifications, custom launch and task allocation mechanics, as well as custom scheduling sources. A simple example would be a simple a process with the capability to execute multiple Tez TaskSpecs as threads. In such a case, a container launch isn't really need and can be mocked. Sourcing / scheduling containers would need to be pluggable. A more advanced example would be LLAP (HIVE-7926; https://issues.apache.org/jira/secure/attachment/12665704/LLAPdesigndocument.pdf). This works with custom interfaces - which would need to be supported by Tez, along with a custom event model which would need translation hooks. Tez should be able to work with a combination of certain vertices running in external services and others running in regular Tez containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2003) [Umbrella] Allow Tez to co-ordinate execution to external services
[ https://issues.apache.org/jira/browse/TEZ-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated TEZ-2003: Attachment: 2003_20150807.1.txt Another patch for jenkins. [Umbrella] Allow Tez to co-ordinate execution to external services -- Key: TEZ-2003 URL: https://issues.apache.org/jira/browse/TEZ-2003 Project: Apache Tez Issue Type: Improvement Reporter: Siddharth Seth Attachments: 2003_20150728.1.txt, 2003_20150807.1.txt, Tez With External Services.pdf The Tez engine itself takes care of co-ordinating execution - controlling how data gets routed (different connection patterns), fault tolerance, scheduling of work, etc. This is currently tied to TaskSpecs defined within Tez and on containers launched by Tez itself (TezChild). The proposal is to allow Tez to work with external services instead of just containers launched by Tez. This involves several more pluggable layers to work with alternate Task Specifications, custom launch and task allocation mechanics, as well as custom scheduling sources. A simple example would be a simple a process with the capability to execute multiple Tez TaskSpecs as threads. In such a case, a container launch isn't really need and can be mocked. Sourcing / scheduling containers would need to be pluggable. A more advanced example would be LLAP (HIVE-7926; https://issues.apache.org/jira/secure/attachment/12665704/LLAPdesigndocument.pdf). This works with custom interfaces - which would need to be supported by Tez, along with a custom event model which would need translation hooks. Tez should be able to work with a combination of certain vertices running in external services and others running in regular Tez containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2003) [Umbrella] Allow Tez to co-ordinate execution to external services
[ https://issues.apache.org/jira/browse/TEZ-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated TEZ-2003: Attachment: 2003_20150728.1.txt Diff against master for jenkins. [Umbrella] Allow Tez to co-ordinate execution to external services -- Key: TEZ-2003 URL: https://issues.apache.org/jira/browse/TEZ-2003 Project: Apache Tez Issue Type: Improvement Reporter: Siddharth Seth Attachments: 2003_20150728.1.txt, Tez With External Services.pdf The Tez engine itself takes care of co-ordinating execution - controlling how data gets routed (different connection patterns), fault tolerance, scheduling of work, etc. This is currently tied to TaskSpecs defined within Tez and on containers launched by Tez itself (TezChild). The proposal is to allow Tez to work with external services instead of just containers launched by Tez. This involves several more pluggable layers to work with alternate Task Specifications, custom launch and task allocation mechanics, as well as custom scheduling sources. A simple example would be a simple a process with the capability to execute multiple Tez TaskSpecs as threads. In such a case, a container launch isn't really need and can be mocked. Sourcing / scheduling containers would need to be pluggable. A more advanced example would be LLAP (HIVE-7926; https://issues.apache.org/jira/secure/attachment/12665704/LLAPdesigndocument.pdf). This works with custom interfaces - which would need to be supported by Tez, along with a custom event model which would need translation hooks. Tez should be able to work with a combination of certain vertices running in external services and others running in regular Tez containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
