I change log4j to debug. I found log in timelineserver. 15/07/24 10:31:55 DEBUG mortbay.log: loaded class com.sun.xml.bind.v2.ContextFactory from sun.misc.Launcher$AppClassLoader@7390d1e8 15/07/24 10:31:55 DEBUG mortbay.log: RESPONSE /ws/v1/timeline/ 200 15/07/24 10:31:55 DEBUG mortbay.log: EOF 15/07/24 10:31:55 DEBUG mortbay.log: EXCEPTION java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:197) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) at org.mortbay.io.nio.ChannelEndPoint.fill(ChannelEndPoint.java:132) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:290) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) 15/07/24 10:31:55 DEBUG mortbay.log: EOF 15/07/24 10:31:55 DEBUG mortbay.log: EOF 15/07/24 10:31:55 DEBUG mortbay.log: EXCEPTION java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:197) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) at org.mortbay.io.nio.ChannelEndPoint.fill(ChannelEndPoint.java:132) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:290) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) 15/07/24 10:31:55 DEBUG mortbay.log: EOF 15/07/24 10:31:55 DEBUG mortbay.log: REQUEST /ws/v1/timeline/ on org.mortbay.jetty.HttpConnection@61567070 15/07/24 10:31:55 DEBUG mortbay.log: EXCEPTION java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:197) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) at org.mortbay.io.nio.ChannelEndPoint.fill(ChannelEndPoint.java:132) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:290) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) 15/07/24 10:31:55 DEBUG mortbay.log: sessionManager=org.mortbay.jetty.servlet.HashSessionManager@59b2947f 15/07/24 10:31:55 DEBUG mortbay.log: session=null 15/07/24 10:31:55 DEBUG mortbay.log: EOF 15/07/24 10:31:55 DEBUG mortbay.log: servlet=default 15/07/24 10:31:55 DEBUG mortbay.log: chain=NoCacheFilter->NoCacheFilter->safety->Timeline Authentication Filter->static_user_filter->guice->default 15/07/24 10:31:55 DEBUG mortbay.log: servlet holder=default 15/07/24 10:31:55 DEBUG mortbay.log: call filter NoCacheFilter 15/07/24 10:31:55 DEBUG mortbay.log: call filter NoCacheFilter 15/07/24 10:31:55 DEBUG mortbay.log: call filter safety 15/07/24 10:31:55 DEBUG mortbay.log: call filter Timeline Authentication Filter 15/07/24 10:31:55 DEBUG server.AuthenticationFilter: Request [http://yhd-jqhadoop200.int.yihaodian.com:8188/ws/v1/timeline/] user [tangjijun] authenticated 15/07/24 10:31:55 DEBUG mortbay.log: call filter static_user_filter 15/07/24 10:31:55 DEBUG mortbay.log: call filter guice 15/07/24 10:31:55 DEBUG mortbay.log: RESPONSE /ws/v1/timeline/ 404 15/07/24 10:31:55 DEBUG mortbay.log: EXCEPTION java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:197) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) at org.mortbay.io.nio.ChannelEndPoint.fill(ChannelEndPoint.java:132) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:290) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) 15/07/24 10:31:55 DEBUG mortbay.log: EOF 15/07/24 10:31:55 DEBUG mortbay.log: EXCEPTION java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:197) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) at org.mortbay.io.nio.ChannelEndPoint.fill(ChannelEndPoint.java:132) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:290) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) 15/07/24 10:31:55 DEBUG mortbay.log: EOF 15/07/24 10:31:55 DEBUG mortbay.log: EXCEPTION
[email protected] From: Hitesh Shah Date: 2015-07-22 23:12 To: user Subject: Re: tez use timeline server get exception Hello jjtang, Can you check the timeline daemon logs on yhd-jqhadoop200.int.yihaodian.com to see if there are any errors? From a functional view point, timeline errors are not fatal but this does mean that the Tez UI might not be able to show all progress info due to the failures to write to history. Also, specifically, I would recommend setting "tez.runtime.convert.user-payload.to.history-text” to false due to the additional load that it creates on Timeline. I know that the tez docs mention it setting it to true but I have a pending jira to update them to not mention this setting. There have been quite a few known reports on problems with Timeline scaling to handling a lot of load. There is some current work in progress to alleviate that. Please follow YARN-3942 and TEZ-2628 jira for more info. Additionally, I am not sure if you can patch TimelineServer but the patch from this jira can help improve performance: YARN-3448 thanks — Hitesh On Jul 21, 2015, at 11:15 PM, jjtang <[email protected]> wrote: > > 2015-07-22 14:00:48,439 ERROR [HistoryEventHandlingThread] > impl.TimelineClientImpl: Failed to get the response from the timeline server. > 2015-07-22 14:00:48,439 WARN [HistoryEventHandlingThread] > ats.ATSHistoryLoggingService: Could not handle history events > org.apache.hadoop.yarn.exceptions.YarnException: Failed to get the response > from the timeline server. > at > org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.doPosting(TimelineClientImpl.java:339) > > at > org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.putEntities(TimelineClientImpl.java:301) > > at > org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService.handleEvents(ATSHistoryLoggingService.java:342) > > at > org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService.access$700(ATSHistoryLoggingService.java:52) > > at > org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService$1.run(ATSHistoryLoggingService.java:186) > > at java.lang.Thread.run(Thread.java:745) > > > My enviroment is hadoop 2.6.0-cdh5.4.3 hive 1.1.0-cdh5.4.3 tez0.7. I type > http://yhd-jqhadoop200.int.yihaodian.com:8188/ws/v1/timeline/ in IE .IE > retrun {"About":"Timeline API"}. > My hiveserver2 hive-site.xml as shown in the following XML snippet: > > <property> > <name>hive.server2.authentication</name> > <value>CUSTOM</value> > </property> > > <property> > <name>hive.server2.custom.authentication.class</name> > <value>com.yhd.hive.auth.YhdPasswdAuthenticationProvider</value> > </property> > > <property> > <name>hive.hs2.authfile</name> > <value>/usr/auth.properties</value> > </property> > <property> > <name>tez.lib.uris</name> > <value>${fs.defaultFS}/user/tez/tez-0.7.0.tar.gz</value> > </property> > <property> > <name>hive.user.install.directory</name> > <value>${fs.defaultFS}/user/</value> > </property> > <property> > <name>tez.history.logging.service.class</name> > <value>org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService</value> > > </property> > <property> > <name>tez.runtime.convert.user-payload.to.history-text</name> > <value>true</value> > </property> > <property> > <description>Indicate to clients whether Timeline service is enabled or not. > If enabled, the TimelineClient library used by end-users will post entities > and events to the Timeline server.</description> > <name>yarn.timeline-service.enabled</name> > <value>true</value> > </property> > > <property> > <description>The hostname of the Timeline service web > application.</description> > <name>yarn.timeline-service.hostname</name> > <value>yhd-jqhadoop200.int.yihaodian.com</value> > </property> > > <property> > <description>Enables cross-origin support (CORS) for web services where > cross-origin web response headers are needed. For example, javascript making > a web services request to the timeline server.</description> > <name>yarn.timeline-service.http-cross-origin.enabled</name> > <value>true</value> > </property> > <property> > <name>tez.tez-ui.history-url.base</name> > <value>http://yhd-jqhadoop200.int.yihaodian.com:8088/tezui</value> > </property> > > > > [email protected] > > > <log.txt>
