Hi Hitesh, Bingo! Log Type: syslog_dag_1476593404620_0001_1
Log Upload Time: Sat Oct 15 22:03:47 -0700 2016 Log Length: 75813 Showing 4096 bytes of 75813 total. Click here <http://dwrdevnn1.sv2.trulia.com:19888/jobhistory/logs/dwrdevdn17.sv2.trulia.com:46114/container_1476593404620_0001_01_000001/container_1476593404620_0001_01_000001/dwr/syslog_dag_1476593404620_0001_1/?start=0> for the full log. 6-10-15 21:51:35,970 [WARN] [IPC Server handler 25 on 40353] |app.TezTaskCommunicatorImpl|: Received task heartbeat from unknown container with id: container_1476593404620_0001_01_000050, asking it to die 2016-10-15 21:51:35,972 [WARN] [IPC Server handler 27 on 40353] |app.TezTaskCommunicatorImpl|: Received task heartbeat from unknown container with id: container_1476593404620_0001_01_000008, asking it to die 2016-10-15 21:51:35,973 [WARN] [IPC Server handler 3 on 40353] |app.TezTaskCommunicatorImpl|: Received task heartbeat from unknown container with id: container_1476593404620_0001_01_000007, asking it to die 2016-10-15 21:51:35,974 [WARN] [IPC Server handler 29 on 40353] |app.TezTaskCommunicatorImpl|: Received task heartbeat from unknown container with id: container_1476593404620_0001_01_000011, asking it to die 2016-10-15 21:51:35,987 [ERROR] [HistoryEventHandlingThread] |impl.TimelineClientImpl|: Failed to get the response from the timeline server. 2016-10-15 21:51:35,987 [WARN] [HistoryEventHandlingThread] |ats.ATSHistoryLoggingService|: Could not handle history events org.apache.hadoop.yarn.exceptions.YarnException: Failed to get the response from the timeline server. at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.doPosting(TimelineClientImpl.java:339) at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.putEntities(TimelineClientImpl.java:301) at org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService.handleEvents(ATSHistoryLoggingService.java:357) at org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService.access$700(ATSHistoryLoggingService.java:53) at org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService$1.run(ATSHistoryLoggingService.java:190) at java.lang.Thread.run(Thread.java:745) 2016-10-15 21:51:35,987 [WARN] [IPC Server handler 6 on 40353] |app.TezTaskCommunicatorImpl|: Received task heartbeat from unknown container with id: container_1476593404620_0001_01_000058, asking it to die 2016-10-15 21:51:35,989 [WARN] [IPC Server handler 24 on 40353] |app.TezTaskCommunicatorImpl|: Received task heartbeat from unknown container with id: container_1476593404620_0001_01_000051, asking it to die 2016-10-15 21:51:36,021 [ERROR] [HistoryEventHandlingThread] |impl.TimelineClientImpl|: Failed to get the response from the timeline server. 2016-10-15 21:51:36,021 [WARN] [HistoryEventHandlingThread] |ats.ATSHistoryLoggingService|: Could not handle history events org.apache.hadoop.yarn.exceptions.YarnException: Failed to get the response from the timeline server. at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.doPosting(TimelineClientImpl.java:339) at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.putEntities(TimelineClientImpl.java:301) at org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService.handleEvents(ATSHistoryLoggingService.java:357) at org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService.access$700(ATSHistoryLoggingService.java:53) at org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService$1.run(ATSHistoryLoggingService.java:190) at java.lang.Thread.run(Thread.java:745) i'm running the hive cli on host=dwrdevnn1. i updated yarn-site.xml on dwrdevnn1. i restarted the ATS service on dwrdevnn1. sudo -u yarn -- yarn-daemon.sh --config /etc/hadoop/conf start timelineserver netstat is showing 8188 as being alive. i can also telnet to dwrdevnn1 8188. also port 10200 is LISTENing. $ sudo netstat -lanp | grep 31168 tcp 0 0 172.19.103.136:10200 0.0.0.0:* LISTEN 31168/java tcp 0 0 172.19.103.136:8188 0.0.0.0:* LISTEN 31168/java might there be a debug log level i can set on impl.TimelineClientImpl to see what is happening on the connection event? thank you again! Cheers, Stephen. On Sun, Oct 16, 2016 at 9:54 AM, Hitesh Shah <hit...@apache.org> wrote: > Hello Stephen, > > yarn-site.xml needs to be updated wherever the Tez client is used. i.e if > you are using Hive, then wherever you launch the Hive CLI and also where > the HiveServer2 is installed ( HS2 will need a restart ). > > To see if the connection to timeline is/was an issue, please check the > yarn app logs for any Tez application ( the application master logs to be > more specific: syslog_dag* files) to see if there are any > warnings/exceptions being logged related to history event handling. > > thanks > — Hitesh > > > On Oct 15, 2016, at 9:58 PM, Stephen Sprague <sprag...@gmail.com> wrote: > > > > hmm... made that change to yarn-site.xml and retarted the timelineserver > and RM. > > > > $ sudo netstat -lanp | grep 31168 #timelineserver > > > > tcp 0 0 172.19.103.136:10200 0.0.0.0:* > LISTEN 31168/java > > tcp 0 0 172.19.103.136:8188 0.0.0.0:* > LISTEN 31168/java > > tcp 0 0 172.19.103.136:8188 172.19.103.136:45299 > ESTABLISHED 31168/java > > tcp 0 0 172.19.103.136:8188 172.19.103.136:45298 > ESTABLISHED 31168/java > > tcp 0 0 172.19.103.136:8188 172.19.103.136:45322 > ESTABLISHED 31168/java > > tcp 0 0 172.19.103.136:8188 172.19.103.136:45297 > ESTABLISHED 31168/java > > tcp 0 0 172.19.103.136:8188 172.19.103.136:45316 > ESTABLISHED 31168/java > > tcp 0 0 172.19.103.136:8188 172.19.103.136:45318 > ESTABLISHED 31168/java > > tcp 0 0 172.19.103.136:8188 172.19.103.136:45317 > ESTABLISHED 31168/java > > tcp 0 0 172.19.103.136:8188 172.19.103.136:45321 > ESTABLISHED 31168/java > > tcp 0 0 172.19.103.136:8188 172.19.103.136:45326 > ESTABLISHED 31168/java > > tcp 0 0 172.19.103.136:8188 172.19.103.136:45314 > ESTABLISHED 31168/java > > tcp 0 0 172.19.103.136:8188 172.19.103.136:45315 > ESTABLISHED 31168/java > > tcp 0 0 172.19.103.136:8188 172.19.103.136:45313 > ESTABLISHED 31168/java > > tcp 0 0 172.19.103.136:8188 172.19.103.136:45320 > ESTABLISHED 31168/java > > tcp 0 0 172.19.103.136:8188 172.19.103.136:45324 > ESTABLISHED 31168/java > > tcp 0 0 172.19.103.136:8188 172.19.103.136:45325 > ESTABLISHED 31168/java > > tcp 0 0 172.19.103.136:8188 172.19.103.136:45319 > ESTABLISHED 31168/java > > unix 2 [ ] STREAM CONNECTED 1455259739 31168/java > > unix 2 [ ] STREAM CONNECTED 1455253313 31168/java > > > > > > still no dice though. same error. i only changed yarn-site.xml on the > namenode though. you think i need to copy it to all the datanodes and > restart the NM's too? > > > > any other suggestions? > > > > 'ppreciate the help! > > > > > > Cheers, > > Stephen. > > > > On Sat, Oct 15, 2016 at 8:46 PM, Allan Wilson <wilsoncr...@gmail.com> > wrote: > > Just saw Gopals response...that def needs updating too. > > > > Sent from my iPhone > > > > On Oct 15, 2016, at 9:31 PM, Stephen Sprague <sprag...@gmail.com> wrote: > > > >> thanks guys. lemme answer. > >> > >> Sreenath- > >> 1. yarn.acl.enable = false (ie. i did not set it) > >> 2. this: http://dwrdevnn1.sv2.trulia.com:9766 displays index.html > with an *empty* list > >> > >> Gopal- > >> 3. i'll replace 0.0.0.0 with dwrdevnn1.sv2.trulia.com and see > happens... > >> > >> Allan- > >> 4. yes, metrics are enabled. > >> > >> > >> I'll let you know what happens with Gopal's suggestion. > >> > >> > >> Cheers, > >> Stephen. > >> > >> On Sat, Oct 15, 2016 at 8:20 PM, Allan Wilson <wilsoncr...@gmail.com> > wrote: > >> Are you emitting metrics to the ATS? > >> > >> yarn.timeline-service.enabled=true > >> > >> Sent from my iPhone > >> > >> On Oct 15, 2016, at 8:36 PM, Sreenath Somarajapuram < > ssomarajapu...@hortonworks.com> wrote: > >> > >>> Hi Stephen, > >>> > >>> The error message is coming from ATS, and it says that the application > data is not available. > >>> And yes, tez_application_1476574340629_0001 is a legit value. It can > be considered as the id for Tez application details. > >>> > >>> Please help me with these: > >>> 1. Are you having yarn.acl.enable = true in yarn-site.xml ? > >>> 2. On going to http://dwrdevnn1.sv2.trulia.com:9766 from your browser > window, the UI is supposed to display a list of DAGs. Are you able to view > them? > >>> > >>> Thanks, > >>> Sreenath > >>> > >>> From: Stephen Sprague <sprag...@gmail.com> > >>> Reply-To: "user@tez.apache.org" <user@tez.apache.org> > >>> Date: Sunday, October 16, 2016 at 7:16 AM > >>> To: "user@tez.apache.org" <user@tez.apache.org> > >>> Subject: Tez UI > >>> > >>> hey guys, > >>> i'm having hard time getting the Tez UI to work. I'm sure i'm doing > something wrong but i can't seem to figure out. Here's my scenario. > >>> > >>> 1. i'm using nginx as the webserver. port 9766. using that port > without params correctly displays index.html. (i followed the instructions > on unzipping the war file - that seems ok - i'm using tez-ui2 ) > >>> > >>> > >>> 2. i run a Tez job. It runs fine. > >>> > >>> > >>> 3. i click on the "History" hyperlink in the RM UI at 8088. > >>> > >>> > >>> 4. it attempts to run http://dwrdevnn1.sv2.trulia. > com:8088/proxy/application_1476574340629_0001/#/tez-app/ > application_1476574340629_0001 > >>> > >>> > >>> 5. which yields this error: > >>> > >>> <image.png> > >>> > >>> i see "id: tez_application_1476574340629_0001" is that "tez_" > prefix legit? > >>> > >>> > >>> > >>> 6. the ATS is running on port 8188. I've modified the file > config/configs.env as well: cf. timeline: "http://dwrdevnn1.sv2.trulia. > com:8188", > >>> > >>> > >>> 7. here are those details: > >>> > >>> yarn 29762 1 12 18:10 pts/5 00:00:11 > /usr/lib/jvm/java-8-oracle/jre/bin/java -Dproc_timelineserver -Xmx1000m > -Dhadoop.log.dir=/var/log/hadoop-yarn -Dyarn.log.dir=/var/log/hadoop-yarn > -Dhadoop.log.file=yarn-yarn-timelineserver-dwrdevnn1.log > -Dyarn.log.file=yarn-yarn-timelineserver-dwrdevnn1.log -Dyarn.home.dir= > -Dyarn.id.str=yarn -Dhadoop.root.logger=INFO,RFA > -Dyarn.root.logger=INFO,RFA -Djava.library.path=/usr/lib/hadoop/lib/native > -Dyarn.policy.file=hadoop-policy.xml -Dhadoop.log.dir=/var/log/hadoop-yarn > -Dyarn.log.dir=/var/log/hadoop-yarn > -Dhadoop.log.file=yarn-yarn-timelineserver-dwrdevnn1.log > -Dyarn.log.file=yarn-yarn-timelineserver-dwrdevnn1.log > -Dyarn.home.dir=/usr/lib/hadoop-yarn -Dhadoop.home.dir=/usr/lib/hadoop-yarn > -Dhadoop.root.logger=INFO,RFA -Dyarn.root.logger=INFO,RFA > -Djava.library.path=/usr/lib/hadoop/lib/native -classpath > /etc/hadoop/conf:/etc/hadoop/conf:/etc/hadoop/conf:/usr/ > lib/hadoop/lib/*:/usr/lib/hadoop/.//*:/usr/lib/hadoop- > hdfs/./:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-hdfs/.// > *:/usr/lib/hadoop-yarn/lib/*:/usr/lib/hadoop-yarn/.//*:/usr/ > lib/hadoop-mapreduce/lib/*:/usr/lib/hadoop-mapreduce/.//*: > /opt/pepperdata/lib/*:/usr/lib/hadoop-yarn/.//*:/usr/lib/ > hadoop-yarn/lib/*:/etc/hadoop/conf/timelineserver-config/log4j.properties > org.apache.hadoop.yarn.server.applicationhistoryservice. > ApplicationHistoryServer > >>> > >>> $ sudo netstat -lanp |grep 29762 > >>> tcp 0 0 0.0.0.0:10200 0.0.0.0:* > LISTEN 29762/java > >>> tcp 0 0 0.0.0.0:8188 0.0.0.0:* > LISTEN 29762/java > >>> > >>> > >>> > >>> 8. the configs in yarn-site.xml > >>> <property> > >>> <name>yarn.timeline-service.hostname</name> > >>> <value>0.0.0.0</value> > >>> </property> > >>> <property> > >>> <name>yarn.timeline-service.enabled</name> > >>> <value>true</value> > >>> </property> > >>> <property> > >>> <name>yarn.timeline-service.webapp.address</name> > >>> <value>0.0.0.0:8188</value> > >>> </property> > >>> <property> > >>> <name>yarn.timeline-service.http-cross-origin.enabled</name> > >>> <value>true</value> > >>> </property> > >>> <property> > >>> <name>yarn.resourcemanager.system-metrics-publisher.enabled</name> > >>> <value>true</value> > >>> </property> > >>> > >>> > >>> 9. and tez-site.xml are as follows: > >>> <property> > >>> <description>Enable Tez to use the Timeline Server for History > Logging</description> > >>> <name>tez.history.logging.service.class</name> > >>> <value>org.apache.tez.dag.history.logging.ats. > ATSHistoryLoggingService</value> > >>> </property> > >>> > >>> <!-- port 9766 defined in nginx config file --> > >>> <property> > >>> <description>URL for where the Tez UI is hosted</description> > >>> <name>tez.tez-ui.history-url.base</name> > >>> <value>http://dwrdevnn1.sv2.trulia.com:9766</value> > >>> </property> > >>> > >>> <!-- from tez-ui README.txt --> > >>> <property> > >>> <name>tez.runtime.convert.user-payload.to.history-text</name> > >>> <value>true</value> > >>> <description>Should be enabled to get the configuration options. > If enabled, the config options are set as > >>> userpayload per input/output. > >>> </description> > >>> </property> > >>> > >>> <property> > >>> <name>tez.allow.disabled.timeline-domains</name> > >>> <value>true</value> > >>> </property> > >>> > >>> > >>> > >>> So i don't get it. Any ideas why this fails? > >>> > >>> thanks, > >>> Stephen. > >>> > >>> > >>> <image.png> > >> > > > >
