[
https://issues.apache.org/jira/browse/YARN-4937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15233515#comment-15233515
]
Steve Loughran commented on YARN-4937:
--------------------------------------
full stack is in a response to a GET request
{code}
2016-04-08 19:05:38,143 [853900744@qtp-1268356702-0] WARN
org.apache.hadoop.yarn.webapp.GenericExceptionHandler
(GenericExceptionHandler.java:toResponse(98)) - INTERNAL_SERVER_ERROR
javax.ws.rs.WebApplicationException:
org.apache.hadoop.service.ServiceStateException:
org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: lock
/Users/stevel/Projects/Hortonworks/Projects/sparkwork/spark/yarn-timeline/yarn-timeline-history/target/tmp/integration/leveldb/timelineEntityGroupId_1111_0_application_1111_0000-timeline-cache.ldb/LOCK:
already held by process
at
org.apache.hadoop.yarn.server.timeline.webapp.TimelineWebServices.getEntity(TimelineWebServices.java:167)
at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
at
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:886)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:96)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:615)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:291)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:574)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1243)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: org.apache.hadoop.service.ServiceStateException:
org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: lock
/Users/stevel/Projects//spark/yarn-timeline/yarn-timeline-history/target/tmp/integration/leveldb/timelineEntityGroupId_1111_0_application_1111_0000-timeline-cache.ldb/LOCK:
already held by process
at
org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
at
org.apache.hadoop.yarn.server.timeline.EntityCacheItem.refreshCache(EntityCacheItem.java:81)
at
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getCachedStore(EntityGroupFSTimelineStore.java:811)
at
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getTimelineStoresFromCacheIds(EntityGroupFSTimelineStore.java:734)
at
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getTimelineStoresForRead(EntityGroupFSTimelineStore.java:767)
at
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getEntity(EntityGroupFSTimelineStore.java:842)
at
org.apache.hadoop.yarn.server.timeline.TimelineDataManager.doGetEntity(TimelineDataManager.java:213)
at
org.apache.hadoop.yarn.server.timeline.TimelineDataManager.getEntity(TimelineDataManager.java:201)
at
org.apache.hadoop.yarn.server.timeline.webapp.TimelineWebServices.getEntity(TimelineWebServices.java:157)
... 50 more
Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error:
lock
/Users/stevel/Projects/spark/yarn-timeline/yarn-timeline-history/target/tmp/integration/leveldb/timelineEntityGroupId_1111_0_application_1111_0000-timeline-cache.ldb/LOCK:
already held by process
at
org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)
at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)
at
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore.serviceInit(LevelDBCacheTimelineStore.java:107)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
... 58 more
2016-04-08 19:05:38,144 [ScalaTest-main-running-ContextToHistoryProviderSuite]
DEBUG org.apac
{code}
> EntityGroupFSTimelineStore to close cached leveldb instances in serviceStop()
> -----------------------------------------------------------------------------
>
> Key: YARN-4937
> URL: https://issues.apache.org/jira/browse/YARN-4937
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
> Affects Versions: 2.8.0
> Environment: JUnit test runs with
> Reporter: Steve Loughran
>
> When I enable ATS1.5 and non-summary data, all my JUnit test runs with it
> fail pass if called individually. But if executed together, the 2nd+ service
> fails, key exception text is
> {code}
> org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: lock
> /Users/stevel/Projects/spark/yarn-timeline/yarn-timeline-history/target/tmp/integration/leveldb/timelineEntityGroupId_1111_0_application_1111_0000-timeline-cache.ldb/LOCK:
> already held by process
> {code}
> I believe the problem is that {{EntityGroupFSTimelineStore.cachedLogs}} isn't
> cleaning up its logs in {{serviceStop()}}, and as my tests use the same
> directory and appIds in its test runs, the next attempt to create a cache
> item fails.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
