[
https://issues.apache.org/jira/browse/TEZ-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14697695#comment-14697695
]
Siddharth Seth commented on TEZ-2003:
-------------------------------------
bq. TaskSchedulerEventHandler
getAvailableResource existed before the changes. That would be used to expose
available resources to plugins, typically. However, it isn't exposed anywhere
at the moment.
instantiateScheduler: TEZ-2714
schedulerId etc are there mainly for consistency.
bq. AMContainer unused imports
TEZ-2714
bq. ShuffleHandler
Removing unused methods: TEZ-2714
SSL enhancements are not planned at the moment. This code is primarily for
testing. If we need to, they can be added when required.
Leaving populateHeader as is for now. Consistent with the original code in
hadoop.
Removing *ChunkedFile. - TEZ-2714 IndexRecord leads to another import which I'd
like to avoid.
bq. Ordered/Unordered Fetchers: dagId
The dagId can be used in a custom shuffle handler to look up a specific
directory. How it's used in LLAP is to write data out to a dag specific
directory, from where the ShuffleHandler serves it. After a query completes,
this entire directory is removed. We need to do the same for Tez at some point
(which involves modifying the YARN/MR ShuffleHandler or introducing our own).
bq. TezClient
This will come in as part of the next rebase
bq. TezTaskRunner
Didn't quite understand the question.
bq. TezTaskRunner2
That shouldn't happen. An interrupt is only sent from TezTaskRunner2 to
TaskRunner2Callable. Before sending this TezTaskRunner2 records the reason for
sending this interrupt, and will use this reason for reporting purposes, rather
than what is returned by TaskRunner2Callable.
> [Umbrella] Allow Tez to co-ordinate execution to external services
> ------------------------------------------------------------------
>
> Key: TEZ-2003
> URL: https://issues.apache.org/jira/browse/TEZ-2003
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Siddharth Seth
> Attachments: 2003_20150728.1.txt, 2003_20150807.1.txt,
> 2003_20150807.2.txt, 2003_20150812.1.txt, 2003_20150812.2.txt, Tez With
> External Services.pdf
>
>
> The Tez engine itself takes care of co-ordinating execution - controlling how
> data gets routed (different connection patterns), fault tolerance, scheduling
> of work, etc.
> This is currently tied to TaskSpecs defined within Tez and on containers
> launched by Tez itself (TezChild).
> The proposal is to allow Tez to work with external services instead of just
> containers launched by Tez. This involves several more pluggable layers to
> work with alternate Task Specifications, custom launch and task allocation
> mechanics, as well as custom scheduling sources.
> A simple example would be a simple a process with the capability to execute
> multiple Tez TaskSpecs as threads. In such a case, a container launch isn't
> really need and can be mocked. Sourcing / scheduling containers would need to
> be pluggable.
> A more advanced example would be LLAP (HIVE-7926;
> https://issues.apache.org/jira/secure/attachment/12665704/LLAPdesigndocument.pdf).
> This works with custom interfaces - which would need to be supported by Tez,
> along with a custom event model which would need translation hooks.
> Tez should be able to work with a combination of certain vertices running in
> external services and others running in regular Tez containers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
