[
https://issues.apache.org/jira/browse/TEZ-3805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated TEZ-3805:
----------------------------------
Attachment: TEZ-3805.1.patch
{noformat}
java -cp `hadoop classpath`:./*:
org.apache.tez.analyzer.plugins.OneOnOneEdgeAnalyzer --saveResults --outputDir
/tmp/test/ --fromSimpleHistory --eventFileName /tmp/q95_1_1_reducer_13.txt
--dagId dag_1490656001509_6107_2
sourceVertex,downstreamVertex,srcTaskId,srcContainerHost,destContainerHost
Reducer 12,Reducer 13,0,mc118-10,mc119-10
Reducer 12,Reducer 13,1,mc118-10,mc120-10
Reducer 12,Reducer 13,3,mc108-10,mc119-10
Reducer 12,Reducer 13,5,mc107-10,mc106-10
Reducer 12,Reducer 13,6,mc119-10,mc107-10
Reducer 12,Reducer 13,7,mc120-10,mc119-10
...
...
Reducer 12,Reducer 13,1003,mc120-10,mc107-10
Reducer 12,Reducer 13,1005,mc107-10,mc106-10
Reducer 12,Reducer 13,1006,mc106-10,mc118-10
Reducer 12,Reducer 13,1008,mc120-10,mc107-10
{noformat}
In this case, it turned out that ~800 tasks out of 1009 tasks were assigned in
different machines.
> Analyzer: Add an analyzer to find out scheduling misses in 1:1 edges
> --------------------------------------------------------------------
>
> Key: TEZ-3805
> URL: https://issues.apache.org/jira/browse/TEZ-3805
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Attachments: TEZ-3805.1.patch
>
>
> When 1:1 edge is used, it would be helpful to find out whether downstream
> tasks ran on the same location provided in the hints by the runtime.
> One of the recent feature in upstream project (hive) used 1:1 edge. Instead
> of checking the logs, it would be useful to have an analyzer to churn out the
> details.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
