[
https://issues.apache.org/jira/browse/TEZ-2888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957445#comment-14957445
]
Bikas Saha commented on TEZ-2888:
---------------------------------
bq. So it should be fine to directly check for getLastDataEvents().size()-1.
Is that a comment that needs to be addressed or an observation? The code is
using that directly right now assuming the asc ordering.
bq. concurrencyByTime is populated, but considered for later use?.
Tooltip/Notes section could be populated with concurrency details?
Yes. its populated for later use. E.g. drawing the concurrency underneath the
CP on the same timeline. That can show more clearly if there was high
concurrency during a long allocation wait period. Please suggest/open jiras for
any other use cases that you may find.
bq. In CriticalPathAnalyzer, maxConcurrency is determined at the DAG level.
But it is quite possible that the initial concurrency was low and gradually it
got more containers. Would the following code break in such cases?
Not sure. The calculation of numWaves would not crash but may be off because we
are using maxConcurrency from some other time period. Perhaps we can use
concurrencyByTime to determine expected concurrency during that time period?
Any other ideas?
Given the non-deterministic nature of YARN scheduling policy, concurrency can
change depending on the state of the cluster and other jobs. So figuring that
out will probably never be accurate.
> Make critical path calculation resilient to AM crash
> ----------------------------------------------------
>
> Key: TEZ-2888
> URL: https://issues.apache.org/jira/browse/TEZ-2888
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: TEZ-2888.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
