[
https://issues.apache.org/jira/browse/TEZ-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14696802#comment-14696802
]
Rajesh Balamohan commented on TEZ-2003:
---------------------------------------
TaskSchedulerEventHandler
- What is the purpose of getAvailableResources?
- Typo in instantiateScheduelrs
- Can schedulerId, task be removed from containerCompleted?. Or is it just
added for consistency?
- Same for containerBeingReleased(), setApplicationRegistrationData(),
preemptContainer()
AMContainer
- Ununsed imports
ShuffleHandler
- ShuffleMetrics metrics context should change? Or remove it if it is not being
used?
- Remove unwanted
serializeMetaData/deserializeMetaData/serializeServiceData/deserializeServiceData
- Will SSL enhancements be in separate JIRA?
- populateHeader does not make use of HttpRequest? Can it be removed?
- Do we need FadvisedChunkedFile/Region/IndexCache etc in tez land?. IndexCache
seems to mainly return TezIndexRecord instead of IndexRecord & there is no
specific usage of TezIndexRecord.hasData in this codebase. So do we need to
import these?
Ordered/Unordered Fetchers:
- "dag" identifier seems to be added in the fetcher url. But not used in the
ShuffleHandler. Is there any other reason for passing this in both type of
fetchers?
TezClient
- needs to wrap ServiceException in RPCUtil TEZ-2663?
TezTaskRunner
{noformat}
//Interrupt the relevant threads. TaskRunner should be interrupted preferably.
if (isTaskRunning()) {
LOG.info("Interrupting taskRunner=" + taskRunner.getName());
taskRunner.interrupt();
} else {
LOG.info("Interrupting waitingThread=" + waitingThread.getName());
waitingThread.interrupt();
}
{noformat}
Should waitingThread.interrupt() be clubbed with the earlier itself as
{noformat} if (!taskRunning.get() || !task.isRunning()) {noformat} ?
TezTaskRunner2
- If any interrupt happens When TaskRunner2Callable is executing, it would exit
with TaskRunner2CallableResult(null) (assuming task.run() is not throwing back
any xception on interrupts). Subsequently
TezTaskRunner2.processCallableResult() would get called, which might set up the
end reason to SUCCESS instead of ERROR?
> [Umbrella] Allow Tez to co-ordinate execution to external services
> ------------------------------------------------------------------
>
> Key: TEZ-2003
> URL: https://issues.apache.org/jira/browse/TEZ-2003
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Siddharth Seth
> Attachments: 2003_20150728.1.txt, 2003_20150807.1.txt,
> 2003_20150807.2.txt, 2003_20150812.1.txt, 2003_20150812.2.txt, Tez With
> External Services.pdf
>
>
> The Tez engine itself takes care of co-ordinating execution - controlling how
> data gets routed (different connection patterns), fault tolerance, scheduling
> of work, etc.
> This is currently tied to TaskSpecs defined within Tez and on containers
> launched by Tez itself (TezChild).
> The proposal is to allow Tez to work with external services instead of just
> containers launched by Tez. This involves several more pluggable layers to
> work with alternate Task Specifications, custom launch and task allocation
> mechanics, as well as custom scheduling sources.
> A simple example would be a simple a process with the capability to execute
> multiple Tez TaskSpecs as threads. In such a case, a container launch isn't
> really need and can be mocked. Sourcing / scheduling containers would need to
> be pluggable.
> A more advanced example would be LLAP (HIVE-7926;
> https://issues.apache.org/jira/secure/attachment/12665704/LLAPdesigndocument.pdf).
> This works with custom interfaces - which would need to be supported by Tez,
> along with a custom event model which would need translation hooks.
> Tez should be able to work with a combination of certain vertices running in
> external services and others running in regular Tez containers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
