-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39648/
-----------------------------------------------------------
Review request for Ambari and Vitalyi Brodetskyi.
Bugs: AMBARI-13558
https://issues.apache.org/jira/browse/AMBARI-13558
Repository: ambari
Description
-------
App Timeline Server unexpectedly turn down
reproduced only for one test run
{code}2015-09-29 15:16:53,742 INFO impl.MetricsConfig
(MetricsConfig.java:loadFirst(112)) - loaded properties from
hadoop-metrics2.properties
2015-09-29 15:16:53,931 INFO impl.MetricsSystemImpl
(MetricsSystemImpl.java:startTimer(377)) - Scheduled snapshot period at 60
second(s).
2015-09-29 15:16:53,931 INFO impl.MetricsSystemImpl
(MetricsSystemImpl.java:start(192)) - ApplicationHistoryServer metrics system
started
2015-09-29 15:16:54,185 INFO timeline.LeveldbTimelineStore
(LeveldbTimelineStore.java:serviceInit(228)) - Using leveldb path
/grid/0/hadoop/yarn/timeline/leveldb-timeline-store.ldb
2015-09-29 15:16:54,196 INFO service.AbstractService
(AbstractService.java:noteFailure(272)) - Service
org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore failed in state
INITED; cause: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO
error: lock /grid/0/hadoop/yarn/timeline/leveldb-timeline-store.ldb/LOCK:
Resource temporarily unavailable
org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: lock
/grid/0/hadoop/yarn/timeline/leveldb-timeline-store.ldb/LOCK: Resource
temporarily unavailable
at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)
at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)
at
org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore.serviceInit(LeveldbTimelineStore.java:229)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceInit(ApplicationHistoryServer.java:103)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.launchAppHistoryServer(ApplicationHistoryServer.java:161)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.main(ApplicationHistoryServer.java:171)
2015-09-29 15:16:54,199 INFO service.AbstractService
(AbstractService.java:noteFailure(272)) - Service
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer
failed in state INITED; cause:
org.apache.hadoop.service.ServiceStateException:
org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: lock
/grid/0/hadoop/yarn/timeline/leveldb-timeline-store.ldb/LOCK: Resource
temporarily unavailable
org.apache.hadoop.service.ServiceStateException:
org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: lock
/grid/0/hadoop/yarn/timeline/leveldb-timeline-store.ldb/LOCK: Resource
temporarily unavailable
at
org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceInit(ApplicationHistoryServer.java:103)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.launchAppHistoryServer(ApplicationHistoryServer.java:161)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.main(ApplicationHistoryServer.java:171)
Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error:
lock /grid/0/hadoop/yarn/timeline/leveldb-timeline-store.ldb/LOCK: Resource
temporarily unavailable
at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)
at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)
at
org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore.serviceInit(LeveldbTimelineStore.java:229)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
... 5 more
2015-09-29 15:16:54,200 INFO impl.MetricsSystemImpl
(MetricsSystemImpl.java:stop(211)) - Stopping ApplicationHistoryServer metrics
system...
2015-09-29 15:16:54,201 INFO impl.MetricsSystemImpl
(MetricsSystemImpl.java:stop(217)) - ApplicationHistoryServer metrics system
stopped.
2015-09-29 15:16:54,201 INFO impl.MetricsSystemImpl
(MetricsSystemImpl.java:shutdown(606)) - ApplicationHistoryServer metrics
system shutdown complete.
2015-09-29 15:16:54,201 FATAL
applicationhistoryservice.ApplicationHistoryServer
(ApplicationHistoryServer.java:launchAppHistoryServer(164)) - Error starting
ApplicationHistoryServer
org.apache.hadoop.service.ServiceStateException:
org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: lock
/grid/0/hadoop/yarn/timeline/leveldb-timeline-store.ldb/LOCK: Resource
temporarily unavailable
at
org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceInit(ApplicationHistoryServer.java:103)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.launchAppHistoryServer(ApplicationHistoryServer.java:161)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.main(ApplicationHistoryServer.java:171)
Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error:
lock /grid/0/hadoop/yarn/timeline/leveldb-timeline-store.ldb/LOCK: Resource
temporarily unavailable
at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)
at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)
at
org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore.serviceInit(LeveldbTimelineStore.java:229)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
... 5 more
2015-09-29 15:16:54,203 INFO util.ExitUtil (ExitUtil.java:terminate(124)) -
Exiting with status -1
2015-09-29 15:16:54,206 INFO
applicationhistoryservice.ApplicationHistoryServer (LogAdapter.java:info(45)) -
SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down ApplicationHistoryServer at
ambari-us-roll-maint-sle113-4235-split3-3/172.22.122.12
************************************************************/
2015-09-29 15:18:30,769 INFO timeline.LeveldbTimelineStore
(LeveldbTimelineStore.java:discardOldEntities(1532)) - Discarded 0 entities for
timestamp 1440861510769 and earlier in 0.0 seconds
2015-09-29 15:23:30,771 INFO timeline.LeveldbTimelineStore
(LeveldbTimelineStore.java:discardOldEntities(1532)) - Discarded 0 entities for
timestamp 1440861810770 and earlier in 0.0 seconds
{code}
Diffs
-----
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
929269d
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/service.py
f368bd4
ambari-server/src/test/python/stacks/2.1/YARN/test_apptimelineserver.py
0e467d8
Diff: https://reviews.apache.org/r/39648/diff/
Testing
-------
mvn clean test
Thanks,
Dmitro Lisnichenko
