[
https://issues.apache.org/jira/browse/TEZ-2004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14680692#comment-14680692
]
Siddharth Seth commented on TEZ-2004:
-------------------------------------
bq. Is ContainerOp necessary ? It seems ContainerLauncherOperationBase can be
used instead, just need to move OPType into ContainerLauncherOperationBase, how
about rename it as ContainerOperationBase ?
ContainerOp is an internal implementation for the regular Tez container
launcher. ContainerLauncherOperationBase is there to separate common
functionality between ContainerStartRequest and ContainerStopRequest in the
API. Exposing OpType would make the API confusing - two ways to figure out the
operation - by the classname and second by the OpType.
bq. TaskCommunicator.java Rename unregisterRunningTaskAttempt to
registerTaskAttemptEnd (make it more consistent)
Will do. This is captured in the TaskComm enhancements jira. Will track in
TEZ-2678 as well.
bq. Rename ContainerLauncherImpl to TezContainerLauncherImpl ? Make all the
default implementation with prefix Tez ?
Will do. Likely post merge or just before the merge to avoid merge conflicts.
Tracking in TEZ-2678
bq. Typo DagTypeConverters.convertServicePluginDescriptoToProto -->
DagTypeConverters.convertServicePluginDescriptorToProto (miss "r")
Tracking in TEZ-2678
bq. Need to verify the DAG's defaultExecutionContext and Vertex's
ExecutionContext exist in the TezClient.servicePluginsDescriptor when
submitting the dag
bq. Need to verify VertexExecutionContext's executeInAm & executeInContainers
is supported in TezClient's ServicePluginsDescriptor
Will add these checks. This is early cross verification to fail early on the
client ? Tracking in TEZ-2678
bq. Seems currently there's only programmatic way to specify TaskScheduler,
ContainerLauncher, TaskCommunicator through TezClient's
ServicePluginsDescriptor, is it expected ? That would mean if some third party
want to introduce new external service to hive, not only they need to implement
new TaskScheduler, ContainerLauncher, TaskCommunicator but also need to change
hive code and rebuild.
Yes. This is expected. Tez exposes this as a programmatic interface only. If
Hive wanted to support custom schedulers, they would need to build a mechanism
around this.
bq. ServicePluginsDescriptor
As my understanding, TaskSchduler/TaskCommunicator/ContainerLauncher are used
together, can not be combined arbitrarily. So should we use
ExecutionContextDescriptor to replace the 3 separators descriptors ?
In code - most are represented as NamedEntityDescriptors. In terms of the API,
they are separate - since it's possible to mix and match them. Two versions of
a scheduler could work with the same launcher and task comm. Similarly, the
uber launcher is used for both local mode and uber mode.
bq. TaskAttempt#scheduleTime may need to put into history event
TaskAttemptStartedEvent to be used by Tez-UI
Captured in TEZ-2678
> Define basic interface for pluggable ContainerLaunchers
> -------------------------------------------------------
>
> Key: TEZ-2004
> URL: https://issues.apache.org/jira/browse/TEZ-2004
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Fix For: TEZ-2003
>
> Attachments: TEZ-2004.1.txt
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
