Rajesh Balamohan created TEZ-4143:
-------------------------------------
Summary: Provide an option to disable DAG graph (.dot) generation
for latency sensitive jobs
Key: TEZ-4143
URL: https://issues.apache.org/jira/browse/TEZ-4143
Project: Apache Tez
Issue Type: Improvement
Reporter: Rajesh Balamohan
In latency sensitive jobs, sometimes it takes time to submit the DAG due to
".dot" file creation in busy cluster. It would be good to have
{{tez.generate.debug.artifacts}} cover for {{DAGAppMaster::generateDAGVizFile}}
Pasting some log snippets for later ref:
{noformat}
syslog_dag_1581571406290_3831_10007_post:2020-04-11 06:36:41,911 [INFO] [IPC
Server handler 0 on 38609] |app.DAGAppMaster|: Generating DAG graphviz file,
dagId=dag_1581571406290_3831_10008,
filePath=/grid/1/hadoop/yarn/log/application_1581571406290_3831/container_e06_1581571406290_3831_01_000001/dag_1581571406290_3831_10008.dot
syslog_dag_1581571406290_3831_10007_post:2020-04-11 06:36:42,744 [INFO] [IPC
Server handler 0 on 38609] |common.TezUtilsInternal|: Redirecting log file
based on addend: dag_1581571406290_3831_10008
syslog_dag_1581571406290_3831_10008_post:2020-04-11 06:36:45,377 [INFO] [IPC
Server handler 1 on 38609] |app.DAGAppMaster|: Generating DAG graphviz file,
dagId=dag_1581571406290_3831_10009,
filePath=/grid/1/hadoop/yarn/log/application_1581571406290_3831/container_e06_1581571406290_3831_01_000001/dag_1581571406290_3831_10009.dot
syslog_dag_1581571406290_3831_10008_post:2020-04-11 06:36:45,690 [INFO] [IPC
Server handler 1 on 38609] |common.TezUtilsInternal|: Redirecting log file
based on addend: dag_1581571406290_3831_10009
syslog_dag_1581571406290_3831_10012_post:2020-04-11 06:36:50,451 [INFO] [IPC
Server handler 0 on 38609] |app.DAGAppMaster|: Generating DAG graphviz file,
dagId=dag_1581571406290_3831_10013,
filePath=/grid/1/hadoop/yarn/log/application_1581571406290_3831/container_e06_1581571406290_3831_01_000001/dag_1581571406290_3831_10013.dot
syslog_dag_1581571406290_3831_10012_post:2020-04-11 06:36:51,574 [INFO] [IPC
Server handler 0 on 38609] |common.TezUtilsInternal|: Redirecting log file
based on addend: dag_1581571406290_3831_10013
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
