Perhaps we should consider creating a TEE in the AM and always do SimpleHistoryLogging?
From: Jianfeng (Jeff) Zhang [mailto:[email protected]] Sent: Thursday, May 21, 2015 5:33 PM To: [email protected] Subject: Re: Tez log location? In that case you are using SimpleHistoryLoggingService rather than ATSHistoryLoggingService. SimpleHistoryLoggingService will log all the messages to the container logs. So if you will find something like the following which is the data same as in /ws/v1/timeline/TEZ_DAG_ID. But these data are only for diagnosis, if you are trying to consume these data for offline analysis, I would encourage you to use data from ATS 2015-05-21 18:52:06,245 INFO [Dispatcher thread: Central] history.HistoryEventHandler: [HISTORY][DAG:dag_1432205503669_0001_1][Event:DAG_FINISHED]: dagId=dag_1432205503669_0001_1, startTime=1432205516089, finishTime=1432205526204, timeTaken=10115, status=SUCCEEDED, diagnostics=, counters=Counters: 14, org.apache.tez.common.counters.DAGCounter, NUM_KILLED_TASKS=1, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=3, AM_CPU_MILLISECONDS=0, AM_GC_TIME_MILLIS=0, File System Counters, HDFS_BYTES_READ=0, HDFS_BYTES_WRITTEN=24, HDFS_READ_OPS=6, HDFS_LARGE_READ_OPS=0, HDFS_WRITE_OPS=4, org.apache.tez.common.counters.TaskCounter, NUM_SPECULATIONS=1, GC_TIME_MILLIS=21, COMMITTED_HEAP_BYTES=514850816, OUTPUT_RECORDS=2 Best Regard, Jeff Zhang From: Xiaoyong Zhu <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Friday, May 22, 2015 at 7:14 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Tez log location? Hi, I am wondering if I didn't configure YARN ATS integration, where would be the tez log (I mean the data available in /ws/v1/timeline/TEZ_DAG_ID if there is an integration happening) go to HDFS? Is there a configuration for that? Thanks! Xiaoyong
