[
https://issues.apache.org/jira/browse/TEZ-2981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15049574#comment-15049574
]
Chris K Wensel commented on TEZ-2981:
-------------------------------------
Having a specific API to talk to the ATS would be fine, but we have a couple
issues with ATS we want to bypass:
* we find the ATS extraordinarily unreliable and slow at scale currently (i
hear there are numerous efforts to rebuild it),
* aligned with "the ATS is unreliable/slow", any additional polling a user app
just adds more load to the ATS
* counter data is only available when a task is completed, it is sometimes
useful to see the counter as it unfolds at the task level - a nice to have
as a related issue, we cannot get a vertex ID from the VertexStatus, but must
ask ATS for it (which is required in order to get its children, the tasks in
question).
fwiw, Cascading users are accustomed to having a normalized view of all runtime
data directly via the Cascading API regardless of the underlying platform and
infrastructure, so we are obliged to retain the portability of their apps from
MapReduce to Tez without loss of functionality and/or robustness. MapReduce job
client api's already provide task level data.
> extend DAGClient API: Give consumers access to the task data
> ------------------------------------------------------------
>
> Key: TEZ-2981
> URL: https://issues.apache.org/jira/browse/TEZ-2981
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: André Kelpe
>
> In Cascading we are sub-classing the DAGClientImpl and implementing the
> TimeLineClient interface in order to get to all the task level data. Since
> these are non-stable APIs we would like to have an API on the DAGClient to
> get to that information w/o having to talk to the TimeLineServer ourselves.
> Similar to how the APIs work over in MapReduce, where there is an API to get
> to the TaskAttempt data w/o having to know, how it is stored in the service.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
