[
https://issues.apache.org/jira/browse/YARN-4772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184981#comment-15184981
]
Steve Loughran commented on YARN-4772:
--------------------------------------
tail of the logs.
Looking at the slf4j message, it may be that this has been happening in test
teardown, so really it's that leveldb had been stopped while another thread was
using it: a race condition on teardown.
{code}
2016-03-08 13:11:44,597 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.security.TimelineACLsManager
(TimelineACLsManager.java:checkAccess(106)) - Verifying the access of stevel on
the timeline entity { id: appattempt_1111_0000_000000, type: spark_event_v01 }
2016-03-08 13:11:44,605 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore
(LogInfo.java:doParse(198)) - Adding
appattempt_1111_0000_000000(spark_event_v01) to store
2016-03-08 13:11:46,588 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.security.TimelineACLsManager
(TimelineACLsManager.java:checkAccess(106)) - Verifying the access of stevel on
the timeline entity { id: appattempt_1111_0000_000000, type: spark_event_v01 }
2016-03-08 13:11:46,597 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore
(LogInfo.java:doParse(198)) - Adding
appattempt_1111_0000_000000(spark_event_v01) to store
2016-03-08 13:11:48,126 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.security.TimelineACLsManager
(TimelineACLsManager.java:checkAccess(106)) - Verifying the access of stevel on
the timeline entity { id: appattempt_1111_0000_000000, type: spark_event_v01 }
2016-03-08 13:11:48,136 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore
(LogInfo.java:doParse(198)) - Adding
appattempt_1111_0000_000000(spark_event_v01) to store
2016-03-08 13:11:49,663 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.security.TimelineACLsManager
(TimelineACLsManager.java:checkAccess(106)) - Verifying the access of stevel on
the timeline entity { id: appattempt_1111_0000_000000, type: spark_event_v01 }
2016-03-08 13:11:49,672 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore
(LogInfo.java:doParse(198)) - Adding
appattempt_1111_0000_000000(spark_event_v01) to store
2016-03-08 13:11:51,588 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.security.TimelineACLsManager
(TimelineACLsManager.java:checkAccess(106)) - Verifying the access of stevel on
the timeline entity { id: appattempt_1111_0000_000000, type: spark_event_v01 }
2016-03-08 13:11:51,596 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore
(LogInfo.java:doParse(198)) - Adding
appattempt_1111_0000_000000(spark_event_v01) to store
2016-03-08 13:11:53,164 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.security.TimelineACLsManager
(TimelineACLsManager.java:checkAccess(106)) - Verifying the access of stevel on
the timeline entity { id: appattempt_1111_0000_000000, type: spark_event_v01 }
2016-03-08 13:11:53,174 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore
(LogInfo.java:doParse(198)) - Adding
appattempt_1111_0000_000000(spark_event_v01) to store
2016-03-08 13:11:54,658 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.security.TimelineACLsManager
(TimelineACLsManager.java:checkAccess(106)) - Verifying the access of stevel on
the timeline entity { id: appattempt_1111_0000_000000, type: spark_event_v01 }
2016-03-08 13:11:54,667 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore
(LogInfo.java:doParse(198)) - Adding
appattempt_1111_0000_000000(spark_event_v01) to store
2016-03-08 13:11:56,526 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.security.TimelineACLsManager
(TimelineACLsManager.java:checkAccess(106)) - Verifying the access of stevel on
the timeline entity { id: appattempt_1111_0000_000000, type: spark_event_v01 }
2016-03-08 13:11:56,534 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore
(LogInfo.java:doParse(198)) - Adding
appattempt_1111_0000_000000(spark_event_v01) to store
2016-03-08 13:11:58,239 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.security.TimelineACLsManager
(TimelineACLsManager.java:checkAccess(106)) - Verifying the access of stevel on
the timeline entity { id: appattempt_1111_0000_000000, type: spark_event_v01 }
2016-03-08 13:11:58,266 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore
(LogInfo.java:doParse(198)) - Adding
appattempt_1111_0000_000000(spark_event_v01) to store
2016-03-08 13:11:59,792 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.security.TimelineACLsManager
(TimelineACLsManager.java:checkAccess(106)) - Verifying the access of stevel on
the timeline entity { id: appattempt_1111_0000_000000, type: spark_event_v01 }
2016-03-08 13:11:59,801 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore
(LogInfo.java:doParse(198)) - Adding
appattempt_1111_0000_000000(spark_event_v01) to store
2016-03-08 13:12:01,612 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.security.TimelineACLsManager
(TimelineACLsManager.java:checkAccess(106)) - Verifying the access of stevel on
the timeline entity { id: appattempt_1111_0000_000000, type: spark_event_v01 }
2016-03-08 13:12:01,621 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore
(LogInfo.java:doParse(198)) - Adding
appattempt_1111_0000_000000(spark_event_v01) to store
2016-03-08 13:12:03,294 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.security.TimelineACLsManager
(TimelineACLsManager.java:checkAccess(106)) - Verifying the access of stevel on
the timeline entity { id: appattempt_1111_0000_000000, type: spark_event_v01 }
2016-03-08 13:12:03,308 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore
(LogInfo.java:doParse(198)) - Adding
appattempt_1111_0000_000000(spark_event_v01) to store
2016-03-08 13:12:05,289 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.security.TimelineACLsManager
(TimelineACLsManager.java:checkAccess(106)) - Verifying the access of stevel on
the timeline entity { id: appattempt_1111_0000_000000, type: spark_event_v01 }
2016-03-08 13:12:05,305 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore
(LogInfo.java:doParse(198)) - Adding
appattempt_1111_0000_000000(spark_event_v01) to store
2016-03-08 13:12:07,090 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.security.TimelineACLsManager
(TimelineACLsManager.java:checkAccess(106)) - Verifying the access of stevel on
the timeline entity { id: appattempt_1111_0000_000000, type: spark_event_v01 }
2016-03-08 13:12:07,100 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore
(LogInfo.java:doParse(198)) - Adding
appattempt_1111_0000_000000(spark_event_v01) to store
2016-03-08 13:12:07,483 [ScalaTest-main-running-ScaleSuite] WARN
org.mortbay.log (Slf4jLog.java:warn(76)) - 4 threads could not be stopped
2016-03-08 13:12:07,568 [ScalaTest-main-running-ScaleSuite] INFO
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore
(EntityGroupFSTimelineStore.java:serviceStop(286)) - Stopping
EntityGroupFSTimelineStore
2016-03-08 13:12:07,573 [ScalaTest-main-running-ScaleSuite] INFO
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore
(EntityGroupFSTimelineStore.java:serviceStop(290)) - Waiting for executor to
terminate
2016-03-08 13:12:08,661 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.security.TimelineACLsManager
(TimelineACLsManager.java:checkAccess(106)) - Verifying the access of stevel on
the timeline entity { id: appattempt_1111_0000_000000, type: spark_event_v01 }
2016-03-08 13:12:08,672 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore
(LogInfo.java:doParse(198)) - Adding
appattempt_1111_0000_000000(spark_event_v01) to store
2016-03-08 13:12:10,248 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.security.TimelineACLsManager
(TimelineACLsManager.java:checkAccess(106)) - Verifying the access of stevel on
the timeline entity { id: appattempt_1111_0000_000000, type: spark_event_v01 }
2016-03-08 13:12:10,257 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore
(LogInfo.java:doParse(198)) - Adding
appattempt_1111_0000_000000(spark_event_v01) to store
2016-03-08 13:12:12,027 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.security.TimelineACLsManager
(TimelineACLsManager.java:checkAccess(106)) - Verifying the access of stevel on
the timeline entity { id: appattempt_1111_0000_000000, type: spark_event_v01 }
2016-03-08 13:12:12,035 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore
(LogInfo.java:doParse(198)) - Adding
appattempt_1111_0000_000000(spark_event_v01) to store
2016-03-08 13:12:13,575 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.security.TimelineACLsManager
(TimelineACLsManager.java:checkAccess(106)) - Verifying the access of stevel on
the timeline entity { id: appattempt_1111_0000_000000, type: spark_event_v01 }
2016-03-08 13:12:13,588 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore
(LogInfo.java:doParse(198)) - Adding
appattempt_1111_0000_000000(spark_event_v01) to store
2016-03-08 13:12:15,578 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.security.TimelineACLsManager
(TimelineACLsManager.java:checkAccess(106)) - Verifying the access of stevel on
the timeline entity { id: appattempt_1111_0000_000000, type: spark_event_v01 }
2016-03-08 13:12:15,589 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore
(LogInfo.java:doParse(198)) - Adding
appattempt_1111_0000_000000(spark_event_v01) to store
2016-03-08 13:12:17,576 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.security.TimelineACLsManager
(TimelineACLsManager.java:checkAccess(106)) - Verifying the access of stevel on
the timeline entity { id: appattempt_1111_0000_000000, type: spark_event_v01 }
2016-03-08 13:12:17,576 [ScalaTest-main-running-ScaleSuite] WARN
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore
(EntityGroupFSTimelineStore.java:serviceStop(295)) - Executor did not terminate
2016-03-08 13:12:17,586 [EntityLogPluginWorker #0] DEBUG
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore
(LogInfo.java:doParse(198)) - Adding
appattempt_1111_0000_000000(spark_event_v01) to store
pthread lock: Invalid argument
/bin/sh: line 1: 8196 Abort trap: 6 java -Djava.awt.headless=true
-Djava.io.tmpdir=/Users/stevel/Projects/Hortonworks/Projects/sparkwork/spark-timeline-integration/target/tmp
-Dscale.test.jobs=100000 -Dspark.testing=1 -Dspark.ui.enabled=false
-Dspark.ui.showConsoleProgress=false -Dspark.unsafe.exceptionOnMemoryLeak=true
-Dbasedir=/Users/stevel/Projects/Hortonworks/Projects/sparkwork/spark-timeline-integration
-ea -Xmx3g -Xss4096k -XX:MaxPermSize=512m -XX:ReservedCodeCacheSize=512m
org.scalatest.tools.Runner -R
'/Users/stevel/Projects/Hortonworks/Projects/sparkwork/spark-timeline-integration/target/classes
/Users/stevel/Projects/Hortonworks/Projects/sparkwork/spark-timeline-integration/target/test-classes'
-w org.apache.spark.deploy.history.yarn.integration.ScaleSuite -o -f
/Users/stevel/Projects/Hortonworks/Projects/sparkwork/spark-timeline-integration/target/surefire-reports/SparkTestSuite.txt
-u
/Users/stevel/Projects/Hortonworks/Projects/sparkwork/spark-timeline-integration/target/surefire-reports/.
{code}
> Overloaded leveljb can crash the ATS "pthread lock: Invalid argument"
> ---------------------------------------------------------------------
>
> Key: YARN-4772
> URL: https://issues.apache.org/jira/browse/YARN-4772
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
> Affects Versions: 2.8.0
> Environment: OSX, scala history scale tests; Java 1.7.0_75-b13
> Reporter: Steve Loughran
>
> while running scale tests with a few hundred thousand events attached to a
> single timeline entity, the JVM crashed
> {code}
> pthread lock: Invalid argument
> /bin/sh: line 1: 8196 Abort trap: 6
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
