[
https://issues.apache.org/jira/browse/TEZ-4505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17748801#comment-17748801
]
László Bodor commented on TEZ-4505:
-----------------------------------
with initial implementation I got something like this, testing further:
attached hive's summary for reference
{code}
INFO :
----------------------------------------------------------------------------------------------
INFO : OPERATION DURATION
INFO :
----------------------------------------------------------------------------------------------
INFO : Compile Query 0.56s
INFO : Prepare Plan 0.22s
INFO : Get Query Coordinator (AM) 42.34s
INFO : Submit Plan 0.10s
INFO : Start DAG 0.39s
INFO : Run DAG 14.77s
INFO :
----------------------------------------------------------------------------------------------
INFO :
INFO : Task Execution Summary
INFO :
----------------------------------------------------------------------------------------------
INFO : VERTICES DURATION(ms) CPU_TIME(ms) GC_TIME(ms)
INPUT_RECORDS OUTPUT_RECORDS
INFO :
----------------------------------------------------------------------------------------------
INFO : Map 1 4053.00 0 0
402 0
INFO : Map 2 11653.00 0 0
28,795,812 10,860,945
INFO : Map 6 4053.00 0 0
10,000 366
INFO : Map 7 6081.00 0 0
2,000,000 2,000,000
INFO : Map 8 4053.00 0 0
10,000 366
INFO : Reducer 3 6515.00 0 0
12,861,146 0
INFO : Reducer 4 0.00 0 0
0 0
INFO : Reducer 5 5570.00 0 0
10,860,945 201
INFO :
----------------------------------------------------------------------------------------------
INFO : States - Dag:
INFO : COMMITTING: 6
INFO : INITED: 2
INFO : NEW: 68
INFO : RUNNING: 15136
INFO : States - Task - Map 1:
INFO : NEW: 20
INFO : RUNNING: 4065
INFO : SCHEDULED: 125
INFO : States - Task - Map 2:
INFO : NEW: 1378
INFO : RUNNING: 346302
INFO : SCHEDULED: 69638
INFO : States - Task - Map 6:
INFO : NEW: 25
INFO : RUNNING: 4103
INFO : SCHEDULED: 133
INFO : States - Task - Map 7:
INFO : NEW: 161
INFO : RUNNING: 37490
INFO : SCHEDULED: 936
INFO : States - Task - Map 8:
INFO : NEW: 25
INFO : RUNNING: 4158
INFO : SCHEDULED: 134
INFO : States - Task - Reducer 3:
INFO : NEW: 46893
INFO : RUNNING: 39646
INFO : SCHEDULED: 104
INFO : States - Task - Reducer 4:
INFO : NEW: 14386
INFO : RUNNING: 233
INFO : SCHEDULED: 2
INFO : States - Task - Reducer 5:
INFO : NEW: 46936
INFO : RUNNING: 32647
INFO : SCHEDULED: 72
INFO : States - TaskAttempt - Map 1:
INFO : NEW: 0
INFO : RUNNING: 12126
INFO : START_WAIT: 372
INFO : SUBMITTED: 0
INFO : States - TaskAttempt - Map 2:
INFO : NEW: 360
INFO : RUNNING: 1039704
INFO : START_WAIT: 207309
INFO : SUBMITTED: 0
INFO : States - TaskAttempt - Map 6:
INFO : NEW: 21
INFO : RUNNING: 12306
INFO : START_WAIT: 369
INFO : SUBMITTED: 3
INFO : States - TaskAttempt - Map 7:
INFO : NEW: 177
INFO : RUNNING: 112371
INFO : START_WAIT: 2574
INFO : SUBMITTED: 3
INFO : States - TaskAttempt - Map 8:
INFO : NEW: 27
INFO : RUNNING: 12438
INFO : START_WAIT: 369
INFO : SUBMITTED: 0
INFO : States - TaskAttempt - Reducer 3:
INFO : NEW: 3
INFO : RUNNING: 119082
INFO : START_WAIT: 159
INFO : SUBMITTED: 0
INFO : States - TaskAttempt - Reducer 4:
INFO : NEW: 0
INFO : RUNNING: 696
INFO : START_WAIT: 6
INFO : SUBMITTED: 0
INFO : States - TaskAttempt - Reducer 5:
INFO : NEW: 3
INFO : RUNNING: 97989
INFO : START_WAIT: 159
INFO : SUBMITTED: 0
{code}
> Create counters about time intervals spent in certain states in
> StateMachineTez
> -------------------------------------------------------------------------------
>
> Key: TEZ-4505
> URL: https://issues.apache.org/jira/browse/TEZ-4505
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> I haven't thought about this in detail, but I'm wondering if can create a
> counter that can tell on vertex/dag level how much time task attempts spent
> idle without being assigned to a container. This might be similar to some
> Hive LLAP counters like:
> https://github.com/apache/hive/blob/master/llap-common/src/java/org/apache/hadoop/hive/llap/counters/LlapWmCounters.java
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
