[
https://issues.apache.org/jira/browse/AMBARI-13558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dmitry Lysnichenko updated AMBARI-13558:
----------------------------------------
Component/s: ambari-server
> App Timeline Server unexpectedly turn down [rarely reproduced]
> --------------------------------------------------------------
>
> Key: AMBARI-13558
> URL: https://issues.apache.org/jira/browse/AMBARI-13558
> Project: Ambari
> Issue Type: Bug
> Components: ambari-server
> Reporter: Dmitry Lysnichenko
> Assignee: Dmitry Lysnichenko
> Attachments: AMBARI-13558.patch
>
>
> App Timeline Server unexpectedly turn down
> reproduced only for one test run
> {code}2015-09-29 15:16:53,742 INFO impl.MetricsConfig
> (MetricsConfig.java:loadFirst(112)) - loaded properties from
> hadoop-metrics2.properties
> 2015-09-29 15:16:53,931 INFO impl.MetricsSystemImpl
> (MetricsSystemImpl.java:startTimer(377)) - Scheduled snapshot period at 60
> second(s).
> 2015-09-29 15:16:53,931 INFO impl.MetricsSystemImpl
> (MetricsSystemImpl.java:start(192)) - ApplicationHistoryServer metrics system
> started
> 2015-09-29 15:16:54,185 INFO timeline.LeveldbTimelineStore
> (LeveldbTimelineStore.java:serviceInit(228)) - Using leveldb path
> /grid/0/hadoop/yarn/timeline/leveldb-timeline-store.ldb
> 2015-09-29 15:16:54,196 INFO service.AbstractService
> (AbstractService.java:noteFailure(272)) - Service
> org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore failed in state
> INITED; cause: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO
> error: lock /grid/0/hadoop/yarn/timeline/leveldb-timeline-store.ldb/LOCK:
> Resource temporarily unavailable
> org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: lock
> /grid/0/hadoop/yarn/timeline/leveldb-timeline-store.ldb/LOCK: Resource
> temporarily unavailable
> at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
> at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)
> at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)
> at
> org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore.serviceInit(LeveldbTimelineStore.java:229)
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceInit(ApplicationHistoryServer.java:103)
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.launchAppHistoryServer(ApplicationHistoryServer.java:161)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.main(ApplicationHistoryServer.java:171)
> 2015-09-29 15:16:54,199 INFO service.AbstractService
> (AbstractService.java:noteFailure(272)) - Service
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer
> failed in state INITED; cause:
> org.apache.hadoop.service.ServiceStateException:
> org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: lock
> /grid/0/hadoop/yarn/timeline/leveldb-timeline-store.ldb/LOCK: Resource
> temporarily unavailable
> org.apache.hadoop.service.ServiceStateException:
> org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: lock
> /grid/0/hadoop/yarn/timeline/leveldb-timeline-store.ldb/LOCK: Resource
> temporarily unavailable
> at
> org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceInit(ApplicationHistoryServer.java:103)
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.launchAppHistoryServer(ApplicationHistoryServer.java:161)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.main(ApplicationHistoryServer.java:171)
> Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error:
> lock /grid/0/hadoop/yarn/timeline/leveldb-timeline-store.ldb/LOCK: Resource
> temporarily unavailable
> at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
> at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)
> at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)
> at
> org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore.serviceInit(LeveldbTimelineStore.java:229)
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> ... 5 more
> 2015-09-29 15:16:54,200 INFO impl.MetricsSystemImpl
> (MetricsSystemImpl.java:stop(211)) - Stopping ApplicationHistoryServer
> metrics system...
> 2015-09-29 15:16:54,201 INFO impl.MetricsSystemImpl
> (MetricsSystemImpl.java:stop(217)) - ApplicationHistoryServer metrics system
> stopped.
> 2015-09-29 15:16:54,201 INFO impl.MetricsSystemImpl
> (MetricsSystemImpl.java:shutdown(606)) - ApplicationHistoryServer metrics
> system shutdown complete.
> 2015-09-29 15:16:54,201 FATAL
> applicationhistoryservice.ApplicationHistoryServer
> (ApplicationHistoryServer.java:launchAppHistoryServer(164)) - Error starting
> ApplicationHistoryServer
> org.apache.hadoop.service.ServiceStateException:
> org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: lock
> /grid/0/hadoop/yarn/timeline/leveldb-timeline-store.ldb/LOCK: Resource
> temporarily unavailable
> at
> org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceInit(ApplicationHistoryServer.java:103)
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.launchAppHistoryServer(ApplicationHistoryServer.java:161)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.main(ApplicationHistoryServer.java:171)
> Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error:
> lock /grid/0/hadoop/yarn/timeline/leveldb-timeline-store.ldb/LOCK: Resource
> temporarily unavailable
> at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
> at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)
> at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)
> at
> org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore.serviceInit(LeveldbTimelineStore.java:229)
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> ... 5 more
> 2015-09-29 15:16:54,203 INFO util.ExitUtil (ExitUtil.java:terminate(124)) -
> Exiting with status -1
> 2015-09-29 15:16:54,206 INFO
> applicationhistoryservice.ApplicationHistoryServer (LogAdapter.java:info(45))
> - SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down ApplicationHistoryServer at
> ambari-us-roll-maint-sle113-4235-split3-3/172.22.122.12
> ************************************************************/
> 2015-09-29 15:18:30,769 INFO timeline.LeveldbTimelineStore
> (LeveldbTimelineStore.java:discardOldEntities(1532)) - Discarded 0 entities
> for timestamp 1440861510769 and earlier in 0.0 seconds
> 2015-09-29 15:23:30,771 INFO timeline.LeveldbTimelineStore
> (LeveldbTimelineStore.java:discardOldEntities(1532)) - Discarded 0 entities
> for timestamp 1440861810770 and earlier in 0.0 seconds
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
