[
https://issues.apache.org/jira/browse/TEZ-3958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16522662#comment-16522662
]
Gopal V commented on TEZ-3958:
------------------------------
Let me take a very specific case here and talk about a DAG in this order.
{code}
digraph g {
store_returns -> Map1 -> ReducerJoin1;
store_sales -> Map2 -> ReducerJoin1;
Map1 -> BloomGen1 -> Map2;
}
{code}
for the hive bloom filter impl (where the small side sends a bloom filter to
the bigger side before a shuffle join).
This has many different stable (i.e no deadlocks) ways to mark out priority,
but not all of them are optimal & it is very hard to debug it (i.e Map1 & Map2
might run in different order if they have their vertex ids swapped).
> Add internal vertex priority information into the tez dag.dot debug
> information
> -------------------------------------------------------------------------------
>
> Key: TEZ-3958
> URL: https://issues.apache.org/jira/browse/TEZ-3958
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Gopal V
> Priority: Major
>
> Adding the actual vertex priority as computed by Tez into the debug dag.dot
> file would allows the debugging of task pre-emption issues when the DAG is no
> longer a tree.
> There are pre-emption issues with isomerization of Tez DAGs, where the a
> R-isomer dag with mirror rotation runs at a different speed than the L-isomer
> dag, due to priorities at the same level changing due to the vertex-id order.
> Since the problem is hard to debug through, it would be good to record the
> computed priority in the DAG .dot file in the logging directories.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
