Found that this is fixed in Yarn 2.6 (
https://issues.apache.org/jira/browse/YARN-2804) which seems unrelated.
I now see the following entries in the timelineserver log, but *still
nothing via webui or via rest API*.
14/11/13 02:10:25 DEBUG timeline.TimelineDataManager: Storing the entity {
id: dag_1414391213387_0105_1, type: TEZ_DAG_ID }, JSON-style content:
{"events":
...
Thanks,
Dharmesh
On Thu, Nov 13, 2014 at 3:01 PM, Dharmesh Kakadia <[email protected]>
wrote:
> Thanks Hitesh. I didn't realize that "yarn.timeline-service.webapp.address"
> is also a client side configuration. May be we should document it
> somewhere. HTH.
>
> After setting the above configuration, I am getting the following
> exception in the TLS logs.
>
> 14/11/13 01:17:27 DEBUG timeline.TimelineDataManager: Storing entities: {
> id: vertex_1414391213387_0101_1_02, type: TEZ_VERTEX_ID }
> 14/11/13 01:17:27 DEBUG server.AuthenticationFilter: Request [
> http://cspod317-02:8188/ws/v1/timeline/?user.name=hadoop] triggering
> authentication
> 14/11/13 01:17:27 DEBUG server.AuthenticationFilter: Request [
> http://cspod317-02:8188/ws/v1/timeline/?user.name=hadoop] user [hadoop]
> authenticated
> Nov 13, 2014 1:17:27 AM
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
> buildModelAndSchemas
> SEVERE: Failed to generate the schema for the JAX-B elements
> com.sun.xml.bind.v2.runtime.IllegalAnnotationsException: 2 counts of
> IllegalAnnotationExceptions
> java.util.Map is an interface, and JAXB can't handle interfaces.
> this problem is related to the following location:
> at java.util.Map
> at public java.util.Map
> org.apache.hadoop.yarn.api.records.timeline.TimelineEvent.getEventInfo()
> at
> org.apache.hadoop.yarn.api.records.timeline.TimelineEvent
> at public java.util.List
> org.apache.hadoop.yarn.api.records.timeline.TimelineEntity.getEvents()
> at
> org.apache.hadoop.yarn.api.records.timeline.TimelineEntity
> java.util.Map does not have a no-arg default constructor.
> this problem is related to the following location:
> at java.util.Map
> at public java.util.Map
> org.apache.hadoop.yarn.api.records.timeline.TimelineEvent.getEventInfo()
> at
> org.apache.hadoop.yarn.api.records.timeline.TimelineEvent
> at public java.util.List
> org.apache.hadoop.yarn.api.records.timeline.TimelineEntity.getEvents()
> at
> org.apache.hadoop.yarn.api.records.timeline.TimelineEntity
>
> at
> com.sun.xml.bind.v2.runtime.IllegalAnnotationsException$Builder.check(IllegalAnnotationsException.java:106)
> at
> com.sun.xml.bind.v2.runtime.JAXBContextImpl.getTypeInfoSet(JAXBContextImpl.java:489)
> at
> com.sun.xml.bind.v2.runtime.JAXBContextImpl.<init>(JAXBContextImpl.java:319)
> at
> com.sun.xml.bind.v2.runtime.JAXBContextImpl$JAXBContextBuilder.build(JAXBContextImpl.java:1170)
> at
> com.sun.xml.bind.v2.ContextFactory.createContext(ContextFactory.java:145)
> at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:248)
> at javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:235)
> at javax.xml.bind.ContextFinder.find(ContextFinder.java:432)
> at javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:637)
> at javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:584)
> at
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator.buildModelAndSchemas(WadlGeneratorJAXBGrammarGenerator.java:412)
> at
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator.createExternalGrammar(WadlGeneratorJAXBGrammarGenerator.java:352)
> at
> com.sun.jersey.server.wadl.WadlBuilder.generate(WadlBuilder.java:115)
> at
> com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.getApplication(WadlApplicationContextImpl.java:104)
> at
> com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.getApplication(WadlApplicationContextImpl.java:120)
> at
> com.sun.jersey.server.impl.wadl.WadlMethodFactory$WadlOptionsMethodDispatcher.dispatch(WadlMethodFactory.java:98)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:886)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
> at
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
> at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
> at
> com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:96)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:572)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:542)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1203)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at
> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
> at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>
> Nov 13, 2014 1:17:27 AM
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
> attachTypes
> INFO: Couldn't find JAX-B element for class
> org.apache.hadoop.yarn.server.webapp.dao.AppsInfo
>
> So, TLS is receiving the event but not able to store it due to the above
> error. Is this version issue? I am running Tez 5.1 and our version of
> hadoop derived from 2.5.
>
> Thanks,
> Dharmesh
>
>
> On Thu, Nov 13, 2014 at 12:10 PM, Hitesh Shah <[email protected]> wrote:
>
>> The communication is via HTTP. The code just uses the TimelineClient
>> which by default is configured to connect to “0.0.0.0:8188” which is
>> usually a problem on a multi-node setup :)
>> To configure it, you need to set yarn.timeline-service.webapp.address to
>> the host:port of the timeline server. This is both a client and server-side
>> property in some sense as the client ( Tez ) needs it to connect to the
>> server and the server uses it to decide which port to bind to.
>>
>> I believe the yarn distributed shell application may have some support of
>> sending data to ATS.
>>
>> A question that should have been asked earlier - what version of Tez are
>> you running?
>>
>> — Hitesh
>>
>> On Nov 12, 2014, at 9:30 PM, Dharmesh Kakadia <[email protected]>
>> wrote:
>>
>> > Hey before filing JIRA, I had just one small question.
>> >
>> > How does the tez communicate to the ATS ? via HDFS directory or via
>> some port? Which configuration on both ATS and Tez side controls that ?
>> >
>> > Also, if you know someway to test ATS, without tez, it would help
>> narrow down the problem.
>> >
>> > Thanks,
>> > Dharmesh
>> >
>> > On Tue, Nov 11, 2014 at 9:51 PM, Jonathan Eagles <[email protected]>
>> wrote:
>> > Great, Dharmesh. Make sure to include the web address you are trying
>> > to browse (for example localhost:8188/ws/v1/timeline/TEZ_DAG_ID). That
>> > will help in debugging.
>> >
>> > Jon
>> >
>> > On Tue, Nov 11, 2014 at 10:17 AM, Dharmesh Kakadia <[email protected]>
>> wrote:
>> > > I will file a JIRA with log attached.
>> > >
>> > > Currently I am only looking at the TLS web UI. Planning to do
>> programmatic
>> > > access if this runs.
>> > >
>> > > Thanks,
>> > > Dharmesh
>> > >
>> > > On Tue, Nov 11, 2014 at 9:41 PM, Hitesh Shah <[email protected]>
>> wrote:
>> > >>
>> > >> Hi Dharmesh
>> > >>
>> > >> Would you mind filing a jira and attaching the application logs
>> obtained
>> > >> by running “bin/yarn logs -applicationId <appId>”
>> > >> Also, I am assuming you are setting “tez.am.log.level” to DEBUG to
>> get
>> > >> debug logging?
>> > >>
>> > >> Also, when you query the timeline server, what are the calls that
>> you are
>> > >> making to retrieve the data?
>> > >>
>> > >> thanks
>> > >> — Hitesh
>> > >>
>> > >>
>> > >> On Nov 11, 2014, at 7:41 AM, Dharmesh Kakadia <[email protected]>
>> wrote:
>> > >>
>> > >> > Both timelineserver and tez was running on the same machine, so I
>> doubt
>> > >> > that could be problem.
>> > >> > Anyway, I set the server address to be the hostname and still
>> nothing on
>> > >> > TLS.
>> > >> >
>> > >> > I appreciate your help.
>> > >> >
>> > >> > Thanks,
>> > >> > Dharmesh
>> > >> >
>> > >> > On Tue, Nov 11, 2014 at 8:53 PM, Prakash Ramachandran
>> > >> > <[email protected]> wrote:
>> > >> > Apologies for the confusion. What I meant was the server address
>> cannot
>> > >> > be 0.0.0.0 it should be some IP address the client can reach to.
>> > >> >
>> > >> > On Nov 11, 2014 8:32 PM, "Dharmesh Kakadia" <[email protected]>
>> wrote:
>> > >> > Hi Prakash,
>> > >> >
>> > >> > I have the "yarn.timeline-service.hostname" already set to 0.0.0.0
>> and
>> > >> > after suggestion from Rajesh, I have set
>> tez.history.logging.service.class
>> > >> > also, but TLS is still not showing anything.
>> > >> >
>> > >> > On a side note, is there any simple way to debug this? I am
>> running tez
>> > >> > with DEBUG level and there is no error I could find.
>> > >> >
>> > >> > Thanks,
>> > >> > Dharmesh
>> > >> >
>> > >> > On Tue, Nov 11, 2014 at 8:07 PM, Prakash Ramachandran
>> > >> > <[email protected]> wrote:
>> > >> > you have set the yarn.timeline-service.hostname to 0.0.0.0.
>> > >> > on the server it means bind on all ip addresses available, however
>> from
>> > >> > client its an invalid address.
>> > >> >
>> > >> > also in tez-site you will need to set the
>> > >> > tez.history.logging.service.class property to
>> > >> > org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService
>> > >> >
>> > >> >
>> > >> > On 11/11/14 7:40 PM, Dharmesh Kakadia wrote:
>> > >> >> I had not set that property initially.
>> > >> >> But even after setting that property in the tez-site, nothing is
>> > >> >> turning up in the TLS.
>> > >> >>
>> > >> >> Thanks,
>> > >> >> Dharmesh
>> > >> >>
>> > >> >> On Tue, Nov 11, 2014 at 7:33 PM, Rajesh Balamohan
>> > >> >> <[email protected]> wrote:
>> > >> >> Can you check if you have set "tez.history.logging.service.class"
>> to
>> > >> >> "org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService"
>> ?
>> > >> >>
>> > >> >> ~Rajesh.B
>> > >> >>
>> > >> >> On Tue, Nov 11, 2014 at 5:55 AM, Dharmesh Kakadia <
>> [email protected]>
>> > >> >> wrote:
>> > >> >> Hi,
>> > >> >>
>> > >> >> I am trying to get Tez job statistics through TmielineServer API.
>> > >> >> I have set up build Tez 0.5.1 and I am able to run tez jobs but
>> no data
>> > >> >> is being pushed to the TimelineServer. THe web interface shows
>> nothing.
>> > >> >>
>> > >> >> Here is what I added to the yarn-site.xml
>> > >> >>
>> > >> >> <property>
>> > >> >> <name>yarn.timeline-service.enabled</name>
>> > >> >> <value>true</value>
>> > >> >> </property>
>> > >> >>
>> > >> >> <property>
>> > >> >> <name>yarn.timeline-service.hostname</name>
>> > >> >> <value>0.0.0.0</value>
>> > >> >> </property>
>> > >> >>
>> > >> >> <property>
>> > >> >>
>> > >> >>
>> <name>yarn.timeline-service.generic-application-history.enabled</name>
>> > >> >> <value>true</value>
>> > >> >> </property>
>> > >> >>
>> > >> >> <property>
>> > >> >>
>> > >> >>
>> <name>yarn.timeline-service.generic-application-history.store-class</name>
>> > >> >>
>> > >> >>
>> <value>org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore</value>
>> > >> >> </property>
>> > >> >>
>> > >> >>
>> > >> >> <property>
>> > >> >> <name>yarn.timeline-service.store-class</name>
>> > >> >>
>> > >> >>
>> <value>org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore</value>
>> > >> >> </property>
>> > >> >>
>> > >> >> Here is my tez-site.xml
>> > >> >>
>> > >> >> <configuration>
>> > >> >> <property>
>> > >> >> <name>tez.lib.uris</name>
>> > >> >> <value>${fs.default.name}/user/hadoop/tez-apps/</value>
>> > >> >> </property>
>> > >> >>
>> > >> >> <property>
>> > >> >> <name>tez.yarn.ats.enabled</name>
>> > >> >> <value>true</value>
>> > >> >> </property>
>> > >> >>
>> > >> >> </configuration>
>> > >> >>
>> > >> >> I can not figure out what am I missing. Any help would be great.
>> > >> >>
>> > >> >> Thanks,
>> > >> >> Dharmesh
>> > >> >>
>> > >> >>
>> > >> >>
>> > >> >>
>> > >> >> --
>> > >> >> ~Rajesh.B
>> > >> >>
>> > >> >
>> > >> >
>> > >> > CONFIDENTIALITY NOTICE
>> > >> > NOTICE: This message is intended for the use of the individual or
>> entity
>> > >> > to which it is addressed and may contain information that is
>> confidential,
>> > >> > privileged and exempt from disclosure under applicable law. If the
>> reader of
>> > >> > this message is not the intended recipient, you are hereby
>> notified that any
>> > >> > printing, copying, dissemination, distribution, disclosure or
>> forwarding of
>> > >> > this communication is strictly prohibited. If you have received
>> this
>> > >> > communication in error, please contact the sender immediately and
>> delete it
>> > >> > from your system. Thank You.
>> > >> >
>> > >> >
>> > >> > CONFIDENTIALITY NOTICE
>> > >> > NOTICE: This message is intended for the use of the individual or
>> entity
>> > >> > to which it is addressed and may contain information that is
>> confidential,
>> > >> > privileged and exempt from disclosure under applicable law. If the
>> reader of
>> > >> > this message is not the intended recipient, you are hereby
>> notified that any
>> > >> > printing, copying, dissemination, distribution, disclosure or
>> forwarding of
>> > >> > this communication is strictly prohibited. If you have received
>> this
>> > >> > communication in error, please contact the sender immediately and
>> delete it
>> > >> > from your system. Thank You.
>> > >> >
>> > >>
>> > >
>> >
>>
>>
>
