For the timeline daemon: 
export YARN_ROOT_LOGGER=“DEBUG, RFA” or the appropriate appender and then 
restart the timeline server. 
or if you don’t want a restart, try http://timelinehost:8188/logLevel and use 
that if available to set the log level to DEBUG for org.apache.hadoop 

For the Tez AM:
Set tez.am.log.level to DEBUG to get more info from the AM logs which will also 
set log level to debug for the timeline client running inside the AM. 
Depending on which version of Tez in use, you set the log level to something 
like "DEBUG;org.apache.hadoop.ipc=INFO;org.apache.hadoop.security=INFO;” where 
everything except the ipc and security package classes log at DEBUG. You can 
use the same format to set the log level to 
“INFO;timeline.client.package=DEBUG;” 

— Hitesh

> On Oct 16, 2016, at 12:32 PM, Stephen Sprague <sprag...@gmail.com> wrote:
> 
> Hi Hitesh,
> Bingo!
> 
> Log Type: syslog_dag_1476593404620_0001_1
> 
> Log Upload Time: Sat Oct 15 22:03:47 -0700 2016
> 
> Log Length: 75813
> 
> Showing 4096 bytes of 75813 total. Click here for the full log.
> 
> 6-10-15 21:51:35,970 [WARN] [IPC Server handler 25 on 40353] 
> |app.TezTaskCommunicatorImpl|: Received task heartbeat from unknown container 
> with id: container_1476593404620_0001_01_000050, asking it to die
> 2016-10-15 21:51:35,972 [WARN] [IPC Server handler 27 on 40353] 
> |app.TezTaskCommunicatorImpl|: Received task heartbeat from unknown container 
> with id: container_1476593404620_0001_01_000008, asking it to die
> 2016-10-15 21:51:35,973 [WARN] [IPC Server handler 3 on 40353] 
> |app.TezTaskCommunicatorImpl|: Received task heartbeat from unknown container 
> with id: container_1476593404620_0001_01_000007, asking it to die
> 2016-10-15 21:51:35,974 [WARN] [IPC Server handler 29 on 40353] 
> |app.TezTaskCommunicatorImpl|: Received task heartbeat from unknown container 
> with id: container_1476593404620_0001_01_000011, asking it to die
> 2016-10-15 21:51:35,987 [ERROR] [HistoryEventHandlingThread] 
> |impl.TimelineClientImpl|: Failed to get the response from the timeline 
> server.
> 2016-10-15 21:51:35,987 [WARN] [HistoryEventHandlingThread] 
> |ats.ATSHistoryLoggingService|: Could not handle history events
> org.apache.hadoop.yarn.exceptions.YarnException: Failed to get the response 
> from the timeline server.
>       at 
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.doPosting(TimelineClientImpl.java:339)
>       at 
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.putEntities(TimelineClientImpl.java:301)
>       at 
> org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService.handleEvents(ATSHistoryLoggingService.java:357)
>       at 
> org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService.access$700(ATSHistoryLoggingService.java:53)
>       at 
> org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService$1.run(ATSHistoryLoggingService.java:190)
>       at java.lang.Thread.run(Thread.java:745)
> 2016-10-15 21:51:35,987 [WARN] [IPC Server handler 6 on 40353] 
> |app.TezTaskCommunicatorImpl|: Received task heartbeat from unknown container 
> with id: container_1476593404620_0001_01_000058, asking it to die
> 2016-10-15 21:51:35,989 [WARN] [IPC Server handler 24 on 40353] 
> |app.TezTaskCommunicatorImpl|: Received task heartbeat from unknown container 
> with id: container_1476593404620_0001_01_000051, asking it to die
> 2016-10-15 21:51:36,021 [ERROR] [HistoryEventHandlingThread] 
> |impl.TimelineClientImpl|: Failed to get the response from the timeline 
> server.
> 2016-10-15 21:51:36,021 [WARN] [HistoryEventHandlingThread] 
> |ats.ATSHistoryLoggingService|: Could not handle history events
> org.apache.hadoop.yarn.exceptions.YarnException: Failed to get the response 
> from the timeline server.
>       at 
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.doPosting(TimelineClientImpl.java:339)
>       at 
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.putEntities(TimelineClientImpl.java:301)
>       at 
> org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService.handleEvents(ATSHistoryLoggingService.java:357)
>       at 
> org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService.access$700(ATSHistoryLoggingService.java:53)
>       at 
> org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService$1.run(ATSHistoryLoggingService.java:190)
>       at java.lang.Thread.run(Thread.java:745)
> 
> 
> 
> i'm running the hive cli on host=dwrdevnn1.
> 
> i updated yarn-site.xml on dwrdevnn1.
> 
> i restarted the ATS service on dwrdevnn1. sudo -u yarn -- yarn-daemon.sh 
> --config /etc/hadoop/conf  start timelineserver
> 
> netstat is showing 8188 as being alive. i can also telnet to dwrdevnn1 8188.  
> also port 10200 is LISTENing.
> 
> $ sudo netstat -lanp | grep 31168
> tcp        0      0 172.19.103.136:10200    0.0.0.0:*               LISTEN    
>   31168/java
> tcp        0      0 172.19.103.136:8188     0.0.0.0:*               LISTEN    
>   31168/java
> 
> 
> might there be a debug log level i can set on impl.TimelineClientImpl to see 
> what is happening on the connection event?  
> 
> thank you again!
> 
> Cheers,
> Stephen.
> 
> 
> 
> 
> On Sun, Oct 16, 2016 at 9:54 AM, Hitesh Shah <hit...@apache.org> wrote:
> Hello Stephen,
> 
> yarn-site.xml needs to be updated wherever the Tez client is used. i.e if you 
> are using Hive, then wherever you launch the Hive CLI and also where the 
> HiveServer2 is installed ( HS2 will need a restart ).
> 
> To see if the connection to timeline is/was an issue, please check the yarn 
> app logs for any Tez application ( the application master logs to be more 
> specific: syslog_dag* files) to see if there are any warnings/exceptions 
> being logged related to history event handling.
> 
> thanks
> — Hitesh
> 
> > On Oct 15, 2016, at 9:58 PM, Stephen Sprague <sprag...@gmail.com> wrote:
> >
> > hmm... made that change to yarn-site.xml and retarted the timelineserver 
> > and RM.
> >
> > $ sudo netstat -lanp | grep 31168 #timelineserver
> >
> > tcp        0      0 172.19.103.136:10200    0.0.0.0:*               LISTEN  
> >     31168/java
> > tcp        0      0 172.19.103.136:8188     0.0.0.0:*               LISTEN  
> >     31168/java
> > tcp        0      0 172.19.103.136:8188     172.19.103.136:45299    
> > ESTABLISHED 31168/java
> > tcp        0      0 172.19.103.136:8188     172.19.103.136:45298    
> > ESTABLISHED 31168/java
> > tcp        0      0 172.19.103.136:8188     172.19.103.136:45322    
> > ESTABLISHED 31168/java
> > tcp        0      0 172.19.103.136:8188     172.19.103.136:45297    
> > ESTABLISHED 31168/java
> > tcp        0      0 172.19.103.136:8188     172.19.103.136:45316    
> > ESTABLISHED 31168/java
> > tcp        0      0 172.19.103.136:8188     172.19.103.136:45318    
> > ESTABLISHED 31168/java
> > tcp        0      0 172.19.103.136:8188     172.19.103.136:45317    
> > ESTABLISHED 31168/java
> > tcp        0      0 172.19.103.136:8188     172.19.103.136:45321    
> > ESTABLISHED 31168/java
> > tcp        0      0 172.19.103.136:8188     172.19.103.136:45326    
> > ESTABLISHED 31168/java
> > tcp        0      0 172.19.103.136:8188     172.19.103.136:45314    
> > ESTABLISHED 31168/java
> > tcp        0      0 172.19.103.136:8188     172.19.103.136:45315    
> > ESTABLISHED 31168/java
> > tcp        0      0 172.19.103.136:8188     172.19.103.136:45313    
> > ESTABLISHED 31168/java
> > tcp        0      0 172.19.103.136:8188     172.19.103.136:45320    
> > ESTABLISHED 31168/java
> > tcp        0      0 172.19.103.136:8188     172.19.103.136:45324    
> > ESTABLISHED 31168/java
> > tcp        0      0 172.19.103.136:8188     172.19.103.136:45325    
> > ESTABLISHED 31168/java
> > tcp        0      0 172.19.103.136:8188     172.19.103.136:45319    
> > ESTABLISHED 31168/java
> > unix  2      [ ]         STREAM     CONNECTED     1455259739 31168/java
> > unix  2      [ ]         STREAM     CONNECTED     1455253313 31168/java
> >
> >
> > still no dice though.  same error.   i only changed yarn-site.xml on the 
> > namenode though.  you think i need to copy it to all the datanodes and 
> > restart the NM's too?
> >
> > any other suggestions?
> >
> > 'ppreciate the help!
> >
> >
> > Cheers,
> > Stephen.
> >
> > On Sat, Oct 15, 2016 at 8:46 PM, Allan Wilson <wilsoncr...@gmail.com> wrote:
> > Just saw Gopals response...that def needs updating too.
> >
> > Sent from my iPhone
> >
> > On Oct 15, 2016, at 9:31 PM, Stephen Sprague <sprag...@gmail.com> wrote:
> >
> >> thanks guys. lemme answer.
> >>
> >> Sreenath-
> >> 1. yarn.acl.enable = false  (ie. i did not set it)
> >> 2.  this:  http://dwrdevnn1.sv2.trulia.com:9766 displays index.html with 
> >> an *empty* list
> >>
> >> Gopal-
> >> 3. i'll replace 0.0.0.0 with dwrdevnn1.sv2.trulia.com and see happens...
> >>
> >> Allan-
> >> 4. yes, metrics are enabled.
> >>
> >>
> >> I'll let you know what happens with Gopal's suggestion.
> >>
> >>
> >> Cheers,
> >> Stephen.
> >>
> >> On Sat, Oct 15, 2016 at 8:20 PM, Allan Wilson <wilsoncr...@gmail.com> 
> >> wrote:
> >> Are you emitting metrics to the ATS?
> >>
> >> yarn.timeline-service.enabled=true
> >>
> >> Sent from my iPhone
> >>
> >> On Oct 15, 2016, at 8:36 PM, Sreenath Somarajapuram 
> >> <ssomarajapu...@hortonworks.com> wrote:
> >>
> >>> Hi Stephen,
> >>>
> >>> The error message is coming from ATS, and it says that the application 
> >>> data is not available.
> >>> And yes, tez_application_1476574340629_0001 is a legit value. It can be 
> >>> considered as the id for Tez application details.
> >>>
> >>> Please help me with these:
> >>> 1. Are you having yarn.acl.enable = true in yarn-site.xml ?
> >>> 2. On going to http://dwrdevnn1.sv2.trulia.com:9766 from your browser 
> >>> window, the UI is supposed to display a list of DAGs. Are you able to 
> >>> view them?
> >>>
> >>> Thanks,
> >>> Sreenath
> >>>
> >>> From: Stephen Sprague <sprag...@gmail.com>
> >>> Reply-To: "user@tez.apache.org" <user@tez.apache.org>
> >>> Date: Sunday, October 16, 2016 at 7:16 AM
> >>> To: "user@tez.apache.org" <user@tez.apache.org>
> >>> Subject: Tez UI
> >>>
> >>> hey guys,
> >>> i'm having hard time getting the Tez UI to work.  I'm sure i'm doing 
> >>> something wrong but i can't seem to figure out.  Here's my scenario.
> >>>
> >>> 1. i'm using nginx as the webserver. port 9766.   using that port without 
> >>> params correctly displays index.html.  (i followed the instructions on 
> >>> unzipping the war file - that seems ok - i'm using tez-ui2 )
> >>>
> >>>
> >>> 2. i run a Tez job. It runs fine.
> >>>
> >>>
> >>> 3. i click on the "History" hyperlink in the RM UI at 8088.
> >>>
> >>>
> >>> 4. it attempts to run 
> >>> http://dwrdevnn1.sv2.trulia.com:8088/proxy/application_1476574340629_0001/#/tez-app/application_1476574340629_0001
> >>>
> >>>
> >>> 5. which yields this error:
> >>>
> >>> <image.png>
> >>>
> >>> i see "id: tez_application_1476574340629_0001"   is that "tez_" prefix 
> >>> legit?
> >>>
> >>>
> >>>
> >>> 6. the ATS is running on port 8188.  I've modified the file 
> >>> config/configs.env as well: cf. timeline: 
> >>> "http://dwrdevnn1.sv2.trulia.com:8188";,
> >>>
> >>>
> >>> 7. here are those details:
> >>>
> >>> yarn     29762     1 12 18:10 pts/5    00:00:11 
> >>> /usr/lib/jvm/java-8-oracle/jre/bin/java -Dproc_timelineserver -Xmx1000m 
> >>> -Dhadoop.log.dir=/var/log/hadoop-yarn -Dyarn.log.dir=/var/log/hadoop-yarn 
> >>> -Dhadoop.log.file=yarn-yarn-timelineserver-dwrdevnn1.log 
> >>> -Dyarn.log.file=yarn-yarn-timelineserver-dwrdevnn1.log -Dyarn.home.dir= 
> >>> -Dyarn.id.str=yarn -Dhadoop.root.logger=INFO,RFA 
> >>> -Dyarn.root.logger=INFO,RFA 
> >>> -Djava.library.path=/usr/lib/hadoop/lib/native 
> >>> -Dyarn.policy.file=hadoop-policy.xml 
> >>> -Dhadoop.log.dir=/var/log/hadoop-yarn -Dyarn.log.dir=/var/log/hadoop-yarn 
> >>> -Dhadoop.log.file=yarn-yarn-timelineserver-dwrdevnn1.log 
> >>> -Dyarn.log.file=yarn-yarn-timelineserver-dwrdevnn1.log 
> >>> -Dyarn.home.dir=/usr/lib/hadoop-yarn 
> >>> -Dhadoop.home.dir=/usr/lib/hadoop-yarn -Dhadoop.root.logger=INFO,RFA 
> >>> -Dyarn.root.logger=INFO,RFA 
> >>> -Djava.library.path=/usr/lib/hadoop/lib/native -classpath 
> >>> /etc/hadoop/conf:/etc/hadoop/conf:/etc/hadoop/conf:/usr/lib/hadoop/lib/*:/usr/lib/hadoop/.//*:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-hdfs/.//*:/usr/lib/hadoop-yarn/lib/*:/usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-mapreduce/lib/*:/usr/lib/hadoop-mapreduce/.//*:/opt/pepperdata/lib/*:/usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-yarn/lib/*:/etc/hadoop/conf/timelineserver-config/log4j.properties
> >>>  
> >>> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer
> >>>
> >>> $ sudo netstat -lanp |grep 29762
> >>> tcp        0      0 0.0.0.0:10200           0.0.0.0:*               
> >>> LISTEN      29762/java
> >>> tcp        0      0 0.0.0.0:8188            0.0.0.0:*               
> >>> LISTEN      29762/java
> >>>
> >>>
> >>>
> >>> 8. the configs in yarn-site.xml
> >>>   <property>
> >>>     <name>yarn.timeline-service.hostname</name>
> >>>     <value>0.0.0.0</value>
> >>>   </property>
> >>>   <property>
> >>>     <name>yarn.timeline-service.enabled</name>
> >>>     <value>true</value>
> >>>   </property>
> >>>   <property>
> >>>     <name>yarn.timeline-service.webapp.address</name>
> >>>     <value>0.0.0.0:8188</value>
> >>>   </property>
> >>>   <property>
> >>>     <name>yarn.timeline-service.http-cross-origin.enabled</name>
> >>>     <value>true</value>
> >>>   </property>
> >>>   <property>
> >>>     <name>yarn.resourcemanager.system-metrics-publisher.enabled</name>
> >>>     <value>true</value>
> >>>   </property>
> >>>
> >>>
> >>> 9. and tez-site.xml are as follows:
> >>>   <property>
> >>>     <description>Enable Tez to use the Timeline Server for History 
> >>> Logging</description>
> >>>     <name>tez.history.logging.service.class</name>
> >>>     
> >>> <value>org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService</value>
> >>>   </property>
> >>>
> >>> <!-- port 9766 defined in nginx config file -->
> >>>   <property>
> >>>     <description>URL for where the Tez UI is hosted</description>
> >>>     <name>tez.tez-ui.history-url.base</name>
> >>>     <value>http://dwrdevnn1.sv2.trulia.com:9766</value>
> >>>   </property>
> >>>
> >>> <!--  from tez-ui README.txt -->
> >>>   <property>
> >>>     <name>tez.runtime.convert.user-payload.to.history-text</name>
> >>>     <value>true</value>
> >>>     <description>Should be enabled to get the configuration options. If 
> >>> enabled, the config options are set as
> >>>                  userpayload per input/output.
> >>>     </description>
> >>>   </property>
> >>>
> >>>    <property>
> >>>       <name>tez.allow.disabled.timeline-domains</name>
> >>>       <value>true</value>
> >>>    </property>
> >>>
> >>>
> >>>
> >>> So i don't get it.  Any ideas why this fails?
> >>>
> >>> thanks,
> >>> Stephen.
> >>>
> >>>
> >>> <image.png>
> >>
> >
> 
> 

Reply via email to