[
https://issues.apache.org/jira/browse/TEZ-3688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated TEZ-3688:
----------------------------------
Attachment: TEZ-3688.1.patch
Attaching .1 patch for review.
For every vertex, it provides "vertexName", "node", "numOfTasks", "totalTasks",
"percentage". This would be useful for quickly checking if there were any
allocation skew of tasks on specific nodes for every vertex.
Example console output is given below.. In zeppelin, this can be a separate
table.
{noformat}
DEBUG [main] (TezAnalyzerBase.java:204) -
--------------------------------------------------------------------------------
DEBUG [main] (TezAnalyzerBase.java:205) - vertexName node
numOfTasks totalTasks Percentage
DEBUG [main] (TezAnalyzerBase.java:206) -
--------------------------------------------------------------------------------
DEBUG [main] (TezAnalyzerBase.java:211) - Map 8 node_n05 1
1 100
DEBUG [main] (TezAnalyzerBase.java:211) - Map 5 node_n05 1
1 100
DEBUG [main] (TezAnalyzerBase.java:211) - Map 6 node_n06 73
697 10.47
DEBUG [main] (TezAnalyzerBase.java:211) - Map 6 node_n09 71
697 10.19
DEBUG [main] (TezAnalyzerBase.java:211) - Map 6 node_n03 71
697 10.19
DEBUG [main] (TezAnalyzerBase.java:211) - Map 6 node_n04 71
697 10.19
DEBUG [main] (TezAnalyzerBase.java:211) - Map 6 node_n08 71
697 10.19
DEBUG [main] (TezAnalyzerBase.java:211) - Map 6 node_n02 70
697 10.04
DEBUG [main] (TezAnalyzerBase.java:211) - Map 6 node_n01 69
697 9.9
DEBUG [main] (TezAnalyzerBase.java:211) - Map 6 node_n10 69
697 9.9
DEBUG [main] (TezAnalyzerBase.java:211) - Map 6 node_n05 67
697 9.61
DEBUG [main] (TezAnalyzerBase.java:211) - Map 6 node_n07 65
697 9.33
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 7 node_n02 269
760 35.39
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 7 node_n06 61
760 8.03
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 7 node_n03 58
760 7.63
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 7 node_n01 56
760 7.37
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 7 node_n05 55
760 7.24
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 7 node_n08 55
760 7.24
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 7 node_n07 53
760 6.97
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 7 node_n04 53
760 6.97
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 7 node_n10 52
760 6.84
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 7 node_n09 48
760 6.32
DEBUG [main] (TezAnalyzerBase.java:211) - Map 1 node_n07 97
910 10.66
DEBUG [main] (TezAnalyzerBase.java:211) - Map 1 node_n09 94
910 10.33
DEBUG [main] (TezAnalyzerBase.java:211) - Map 1 node_n06 94
910 10.33
DEBUG [main] (TezAnalyzerBase.java:211) - Map 1 node_n02 92
910 10.11
DEBUG [main] (TezAnalyzerBase.java:211) - Map 1 node_n05 91
910 10
DEBUG [main] (TezAnalyzerBase.java:211) - Map 1 node_n04 91
910 10
DEBUG [main] (TezAnalyzerBase.java:211) - Map 1 node_n10 91
910 10
DEBUG [main] (TezAnalyzerBase.java:211) - Map 1 node_n03 89
910 9.78
DEBUG [main] (TezAnalyzerBase.java:211) - Map 1 node_n01 88
910 9.67
DEBUG [main] (TezAnalyzerBase.java:211) - Map 1 node_n08 83
910 9.12
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 2 node_n02 565
1009 56
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 2 node_n09 54
1009 5.35
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 2 node_n04 54
1009 5.35
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 2 node_n08 51
1009 5.05
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 2 node_n05 50
1009 4.96
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 2 node_n07 49
1009 4.86
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 2 node_n03 49
1009 4.86
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 2 node_n10 48
1009 4.76
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 2 node_n01 46
1009 4.56
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 2 node_n06 43
1009 4.26
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 3 node_n07 111
1009 11
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 3 node_n06 109
1009 10.8
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 3 node_n08 108
1009 10.7
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 3 node_n10 107
1009 10.6
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 3 node_n02 106
1009 10.51
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 3 node_n05 102
1009 10.11
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 3 node_n04 96
1009 9.51
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 3 node_n03 93
1009 9.22
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 3 node_n01 89
1009 8.82
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 3 node_n09 88
1009 8.72
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 4 node_n03 1
1 100
DEBUG [main] (TezAnalyzerBase.java:213) -
--------------------------------------------------------------------------------
{noformat}
> Add analyzer to provide the set of machines that participated in the vertex
> and the number of tasks that ran on that node
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: TEZ-3688
> URL: https://issues.apache.org/jira/browse/TEZ-3688
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Nita Dembla
> Assignee: Rajesh Balamohan
> Attachments: TEZ-3688.1.patch
>
>
> Need an analyzer to parse DAG info and outputs the how many tasks of a vertex
> ran on a node. This would help in isolating task scheduling issues.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
