[
https://issues.apache.org/jira/browse/TEZ-3688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated TEZ-3688:
----------------------------------
Attachment: TEZ-3688.2.patch
Attaching .2 version with additional 2 metrics.
"firstTaskToLastTaskTime" - time taken from last attempt's finish time to first
attempt's start time for a specific vertex running on same node.
"totalRuntimeOfTaskAttempts" - Overall runtime total for all task attempts
running on same node for every vertex.
{noformat}
DEBUG [main] (TezAnalyzerBase.java:204) -
--------------------------------------------------------------------------------
DEBUG [main] (TezAnalyzerBase.java:205) - vertexName node numOfTasks
totalTasks percentage firstTaskToLastTaskTime totalRuntimeOfTaskAttempts
DEBUG [main] (TezAnalyzerBase.java:206) -
--------------------------------------------------------------------------------
DEBUG [main] (TezAnalyzerBase.java:211) - Map 8 node_n05 1 1
100 49 49
DEBUG [main] (TezAnalyzerBase.java:211) - Map 5 node_n05 1 1
100 49 49
DEBUG [main] (TezAnalyzerBase.java:211) - Map 6 node_n06 73
697 10.47 29095 1120072
DEBUG [main] (TezAnalyzerBase.java:211) - Map 6 node_n09 71
697 10.19 40464 1095435
DEBUG [main] (TezAnalyzerBase.java:211) - Map 6 node_n03 71
697 10.19 38283 1107531
DEBUG [main] (TezAnalyzerBase.java:211) - Map 6 node_n04 71
697 10.19 33980 1104011
DEBUG [main] (TezAnalyzerBase.java:211) - Map 6 node_n08 71
697 10.19 34983 1135480
DEBUG [main] (TezAnalyzerBase.java:211) - Map 6 node_n02 70
697 10.04 32567 1096888
DEBUG [main] (TezAnalyzerBase.java:211) - Map 6 node_n01 69
697 9.9 29274 1087283
DEBUG [main] (TezAnalyzerBase.java:211) - Map 6 node_n10 69
697 9.9 31584 1077462
DEBUG [main] (TezAnalyzerBase.java:211) - Map 6 node_n05 67
697 9.61 18967 1042949
DEBUG [main] (TezAnalyzerBase.java:211) - Map 6 node_n07 65
697 9.33 27261 1009257
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 7 node_n02 269
760 35.39 164403 5989416
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 7 node_n06 61
760 8.03 406732 11894236
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 7 node_n03 58
760 7.63 369869 10751886
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 7 node_n01 56
760 7.37 294363 10901085
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 7 node_n05 55
760 7.24 400042 10744801
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 7 node_n08 55
760 7.24 313695 10802027
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 7 node_n07 53
760 6.97 507109 11147493
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 7 node_n04 53
760 6.97 272674 10363566
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 7 node_n10 52
760 6.84 401464 10897959
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 7 node_n09 48
760 6.32 377443 10321227
DEBUG [main] (TezAnalyzerBase.java:211) - Map 1 node_n07 97
910 10.66 67959 2001824
DEBUG [main] (TezAnalyzerBase.java:211) - Map 1 node_n09 94
910 10.33 56145 1954117
DEBUG [main] (TezAnalyzerBase.java:211) - Map 1 node_n06 94
910 10.33 65015 1837277
DEBUG [main] (TezAnalyzerBase.java:211) - Map 1 node_n02 92
910 10.11 70143 1913512
DEBUG [main] (TezAnalyzerBase.java:211) - Map 1 node_n05 91
910 10 60574 1949792
DEBUG [main] (TezAnalyzerBase.java:211) - Map 1 node_n04 91
910 10 66346 1872313
DEBUG [main] (TezAnalyzerBase.java:211) - Map 1 node_n10 91
910 10 67908 1906150
DEBUG [main] (TezAnalyzerBase.java:211) - Map 1 node_n03 89
910 9.78 69197 1909219
DEBUG [main] (TezAnalyzerBase.java:211) - Map 1 node_n01 88
910 9.67 63391 1889935
DEBUG [main] (TezAnalyzerBase.java:211) - Map 1 node_n08 83
910 9.12 57916 1842694
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 2 node_n02 565
1009 56 346802 13854740
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 2 node_n09 54
1009 5.35 418495 12466793
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 2 node_n04 54
1009 5.35 412465 12533686
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 2 node_n08 51
1009 5.05 375560 10294875
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 2 node_n05 50
1009 4.96 413360 11896801
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 2 node_n07 49
1009 4.86 373335 9874865
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 2 node_n03 49
1009 4.86 418330 12113031
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 2 node_n10 48
1009 4.76 379399 10205819
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 2 node_n01 46
1009 4.56 412900 11819939
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 2 node_n06 43
1009 4.26 371551 9158577
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 3 node_n07 111
1009 11 317096 11627659
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 3 node_n06 109
1009 10.8 309433 11526032
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 3 node_n08 108
1009 10.7 330952 11713705
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 3 node_n10 107
1009 10.6 321248 11672844
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 3 node_n02 106
1009 10.51 278944 10677878
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 3 node_n05 102
1009 10.11 333696 10168223
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 3 node_n04 96
1009 9.51 307277 9303032
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 3 node_n03 93
1009 9.22 315551 8835746
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 3 node_n01 89
1009 8.82 310642 8730957
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 3 node_n09 88
1009 8.72 302418 8488410
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 4 node_n03 1 1
100 120728 120728
DEBUG [main] (TezAnalyzerBase.java:213) -
--------------------------------------------------------------------------------
{noformat}
> Add analyzer to provide the set of machines that participated in the vertex
> and the number of tasks that ran on that node
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: TEZ-3688
> URL: https://issues.apache.org/jira/browse/TEZ-3688
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Nita Dembla
> Assignee: Rajesh Balamohan
> Attachments: TEZ-3688.1.patch, TEZ-3688.2.patch
>
>
> Need an analyzer to parse DAG info and outputs the how many tasks of a vertex
> ran on a node. This would help in isolating task scheduling issues.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
