got it. thanks! On Thu, May 7, 2015 at 2:52 PM, Marcelo Vanzin <van...@cloudera.com> wrote:
> Ah, sorry, that's definitely what Shixiong mentioned. The patch I > mentioned did not make it into 1.3... > > On Thu, May 7, 2015 at 11:48 AM, Koert Kuipers <ko...@tresata.com> wrote: > >> seems i got one thread spinning 100% for a while now, in >> FsHistoryProvider.initialize(). maybe something wrong with my logs on hdfs >> that its reading? or could it simply really take 30 mins to read all the >> history on dhfs? >> >> jstack: >> >> Deadlock Detection: >> >> No deadlocks found. >> >> Thread 2272: (state = BLOCKED) >> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; >> information may be imprecise) >> - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, >> long) @bci=20, line=226 (Compiled frame) >> - >> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, >> boolean, long) @bci=174, line=460 (Compiled frame) >> - >> java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, >> boolean, long) @bci=102, line=359 (Interpreted frame) >> - java.util.concurrent.SynchronousQueue.poll(long, >> java.util.concurrent.TimeUnit) @bci=11, line=942 (Interpreted frame) >> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1068 >> (Interpreted frame) >> - >> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) >> @bci=26, line=1130 (Interpreted frame) >> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 >> (Interpreted frame) >> - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame) >> >> >> Thread 1986: (state = BLOCKED) >> - java.lang.Thread.sleep(long) @bci=0 (Interpreted frame) >> - org.apache.hadoop.hdfs.PeerCache.run() @bci=41, line=250 (Interpreted >> frame) >> - >> org.apache.hadoop.hdfs.PeerCache.access$000(org.apache.hadoop.hdfs.PeerCache) >> @bci=1, line=41 (Interpreted frame) >> - org.apache.hadoop.hdfs.PeerCache$1.run() @bci=4, line=119 (Interpreted >> frame) >> - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame) >> >> >> Thread 1970: (state = BLOCKED) >> >> >> Thread 1969: (state = BLOCKED) >> - java.lang.Object.wait(long) @bci=0 (Interpreted frame) >> - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=135 >> (Interpreted frame) >> - java.lang.ref.ReferenceQueue.remove() @bci=2, line=151 (Interpreted >> frame) >> - java.lang.ref.Finalizer$FinalizerThread.run() @bci=36, line=209 >> (Interpreted frame) >> >> >> Thread 1968: (state = BLOCKED) >> - java.lang.Object.wait(long) @bci=0 (Interpreted frame) >> - java.lang.Object.wait() @bci=2, line=503 (Interpreted frame) >> - java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=133 >> (Interpreted frame) >> >> >> Thread 1958: (state = IN_VM) >> - java.lang.Throwable.fillInStackTrace(int) @bci=0 (Compiled frame; >> information may be imprecise) >> - java.lang.Throwable.fillInStackTrace() @bci=16, line=783 (Compiled >> frame) >> - java.lang.Throwable.<init>(java.lang.String, java.lang.Throwable) >> @bci=24, line=287 (Compiled frame) >> - java.lang.Exception.<init>(java.lang.String, java.lang.Throwable) >> @bci=3, line=84 (Compiled frame) >> - org.json4s.package$MappingException.<init>(java.lang.String, >> java.lang.Exception) @bci=13, line=56 (Compiled frame) >> - org.json4s.reflect.package$.fail(java.lang.String, >> java.lang.Exception) @bci=6, line=96 (Compiled frame) >> - org.json4s.Extraction$.convert(org.json4s.JsonAST$JValue, >> org.json4s.reflect.ScalaType, org.json4s.Formats, scala.Option) @bci=2447, >> line=554 (Compiled frame) >> - org.json4s.Extraction$.extract(org.json4s.JsonAST$JValue, >> org.json4s.reflect.ScalaType, org.json4s.Formats) @bci=796, line=331 >> (Compiled frame) >> - org.json4s.Extraction$.extract(org.json4s.JsonAST$JValue, >> org.json4s.Formats, scala.reflect.Manifest) @bci=10, line=42 (Compiled >> frame) >> - org.json4s.Extraction$.extractOpt(org.json4s.JsonAST$JValue, >> org.json4s.Formats, scala.reflect.Manifest) @bci=7, line=54 (Compiled frame) >> - org.json4s.ExtractableJsonAstNode.extractOpt(org.json4s.Formats, >> scala.reflect.Manifest) @bci=9, line=40 (Compiled frame) >> - >> org.apache.spark.util.JsonProtocol$.shuffleWriteMetricsFromJson(org.json4s.JsonAST$JValue) >> @bci=116, line=702 (Compiled frame) >> - >> org.apache.spark.util.JsonProtocol$$anonfun$taskMetricsFromJson$2.apply(org.json4s.JsonAST$JValue) >> @bci=4, line=670 (Compiled frame) >> - >> org.apache.spark.util.JsonProtocol$$anonfun$taskMetricsFromJson$2.apply(java.lang.Object) >> @bci=5, line=670 (Compiled frame) >> - scala.Option.map(scala.Function1) @bci=22, line=145 (Compiled frame) >> - >> org.apache.spark.util.JsonProtocol$.taskMetricsFromJson(org.json4s.JsonAST$JValue) >> @bci=414, line=670 (Compiled frame) >> - >> org.apache.spark.util.JsonProtocol$.taskEndFromJson(org.json4s.JsonAST$JValue) >> @bci=174, line=508 (Compiled frame) >> - >> org.apache.spark.util.JsonProtocol$.sparkEventFromJson(org.json4s.JsonAST$JValue) >> @bci=389, line=464 (Compiled frame) >> - >> org.apache.spark.scheduler.ReplayListenerBus$$anonfun$replay$1.apply(java.lang.String) >> @bci=34, line=51 (Compiled frame) >> - >> org.apache.spark.scheduler.ReplayListenerBus$$anonfun$replay$1.apply(java.lang.Object) >> @bci=5, line=49 (Compiled frame) >> - scala.collection.Iterator$class.foreach(scala.collection.Iterator, >> scala.Function1) @bci=16, line=743 (Compiled frame) >> - scala.collection.AbstractIterator.foreach(scala.Function1) @bci=2, >> line=1177 (Interpreted frame) >> - >> org.apache.spark.scheduler.ReplayListenerBus.replay(java.io.InputStream, >> java.lang.String) @bci=42, line=49 (Interpreted frame) >> - >> org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$replay(org.apache.hadoop.fs.FileStatus, >> org.apache.spark.scheduler.ReplayListenerBus) @bci=69, line=260 >> (Interpreted frame) >> - >> org.apache.spark.deploy.history.FsHistoryProvider$$anonfun$6.apply(org.apache.hadoop.fs.FileStatus) >> @bci=19, line=190 (Interpreted frame) >> - >> org.apache.spark.deploy.history.FsHistoryProvider$$anonfun$6.apply(java.lang.Object) >> @bci=5, line=188 (Interpreted frame) >> - >> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(java.lang.Object) >> @bci=9, line=252 (Interpreted frame) >> - >> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(java.lang.Object) >> @bci=2, line=252 (Interpreted frame) >> - >> scala.collection.IndexedSeqOptimized$class.foreach(scala.collection.IndexedSeqOptimized, >> scala.Function1) @bci=22, line=33 (Interpreted frame) >> - scala.collection.mutable.WrappedArray.foreach(scala.Function1) @bci=2, >> line=35 (Interpreted frame) >> - >> scala.collection.TraversableLike$class.flatMap(scala.collection.TraversableLike, >> scala.Function1, scala.collection.generic.CanBuildFrom) @bci=17, line=252 >> (Interpreted frame) >> - scala.collection.AbstractTraversable.flatMap(scala.Function1, >> scala.collection.generic.CanBuildFrom) @bci=3, line=104 (Interpreted frame) >> - org.apache.spark.deploy.history.FsHistoryProvider.checkForLogs() >> @bci=110, line=188 (Interpreted frame) >> - org.apache.spark.deploy.history.FsHistoryProvider.initialize() >> @bci=38, line=116 (Interpreted frame) >> - >> org.apache.spark.deploy.history.FsHistoryProvider.<init>(org.apache.spark.SparkConf) >> @bci=214, line=99 (Interpreted frame) >> - >> sun.reflect.NativeConstructorAccessorImpl.newInstance0(java.lang.reflect.Constructor, >> java.lang.Object[]) @bci=0 (Interpreted frame) >> - >> sun.reflect.NativeConstructorAccessorImpl.newInstance(java.lang.Object[]) >> @bci=72, line=57 (Interpreted frame) >> - >> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(java.lang.Object[]) >> @bci=5, line=45 (Interpreted frame) >> - java.lang.reflect.Constructor.newInstance(java.lang.Object[]) @bci=79, >> line=526 (Interpreted frame) >> - >> org.apache.spark.deploy.history.HistoryServer$.main(java.lang.String[]) >> @bci=89, line=185 (Interpreted frame) >> - org.apache.spark.deploy.history.HistoryServer.main(java.lang.String[]) >> @bci=4 (Interpreted frame) >> >> >> >> On Thu, May 7, 2015 at 2:17 PM, Koert Kuipers <ko...@tresata.com> wrote: >> >>> good idea i will take a look. it does seem to be spinning one cpu at >>> 100%... >>> >>> On Thu, May 7, 2015 at 2:03 PM, Marcelo Vanzin <van...@cloudera.com> >>> wrote: >>> >>>> Can you get a jstack for the process? Maybe it's stuck somewhere. >>>> >>>> On Thu, May 7, 2015 at 11:00 AM, Koert Kuipers <ko...@tresata.com> >>>> wrote: >>>> >>>>> i am trying to launch the spark 1.3.1 history server on a secure >>>>> cluster. >>>>> >>>>> i can see in the logs that it successfully logs into kerberos, and it >>>>> is replaying all the logs, but i never see the log message that indicate >>>>> the web server is started (i should see something like "Successfully >>>>> started service on port 18080." or "Started HistoryServer at >>>>> http://somehost:18080";). yet the daemon stays alive... >>>>> >>>>> any idea why the history server would never start the web service? >>>>> >>>>> thanks! >>>>> >>>>> >>>> >>>> >>>> -- >>>> Marcelo >>>> >>> >>> >> > > > -- > Marcelo >
