Thanks everyone. Yiannis, do you know if there's a bug report for this regression? For some other (possibly connected) reason I upgraded from 1.1.1 to 1.2.1, but I can't remember what the bug was.
Joe On 24 February 2015 at 19:26, Yiannis Gkoufas <johngou...@gmail.com> wrote: > Hi there, > > I assume you are using spark 1.2.1 right? > I faced the exact same issue and switched to 1.1.1 with the same > configuration and it was solved. > On 24 Feb 2015 19:22, "Ted Yu" <yuzhih...@gmail.com> wrote: > >> Here is a tool which may give you some clue: >> http://file-leak-detector.kohsuke.org/ >> >> Cheers >> >> On Tue, Feb 24, 2015 at 11:04 AM, Vladimir Rodionov < >> vrodio...@splicemachine.com> wrote: >> >>> Usually it happens in Linux when application deletes file w/o double >>> checking that there are no open FDs (resource leak). In this case, Linux >>> holds all space allocated and does not release it until application >>> exits (crashes in your case). You check file system and everything is >>> normal, you have enough space and you have no idea why does application >>> report "no space left on device". >>> >>> Just a guess. >>> >>> -Vladimir Rodionov >>> >>> On Tue, Feb 24, 2015 at 8:34 AM, Joe Wass <jw...@crossref.org> wrote: >>> >>>> I'm running a cluster of 3 Amazon EC2 machines (small number because >>>> it's expensive when experiments keep crashing after a day!). >>>> >>>> Today's crash looks like this (stacktrace at end of message). >>>> org.apache.spark.shuffle.MetadataFetchFailedException: Missing an >>>> output location for shuffle 0 >>>> >>>> On my three nodes, I have plenty of space and inodes: >>>> >>>> A $ df -i >>>> Filesystem Inodes IUsed IFree IUse% Mounted on >>>> /dev/xvda1 524288 97937 426351 19% / >>>> tmpfs 1909200 1 1909199 1% /dev/shm >>>> /dev/xvdb 2457600 54 2457546 1% /mnt >>>> /dev/xvdc 2457600 24 2457576 1% /mnt2 >>>> /dev/xvds 831869296 23844 831845452 1% /vol0 >>>> >>>> A $ df -h >>>> Filesystem Size Used Avail Use% Mounted on >>>> /dev/xvda1 7.9G 3.4G 4.5G 44% / >>>> tmpfs 7.3G 0 7.3G 0% /dev/shm >>>> /dev/xvdb 37G 1.2G 34G 4% /mnt >>>> /dev/xvdc 37G 177M 35G 1% /mnt2 >>>> /dev/xvds 1000G 802G 199G 81% /vol0 >>>> >>>> B $ df -i >>>> Filesystem Inodes IUsed IFree IUse% Mounted on >>>> /dev/xvda1 524288 97947 426341 19% / >>>> tmpfs 1906639 1 1906638 1% /dev/shm >>>> /dev/xvdb 2457600 54 2457546 1% /mnt >>>> /dev/xvdc 2457600 24 2457576 1% /mnt2 >>>> /dev/xvds 816200704 24223 816176481 1% /vol0 >>>> >>>> B $ df -h >>>> Filesystem Size Used Avail Use% Mounted on >>>> /dev/xvda1 7.9G 3.6G 4.3G 46% / >>>> tmpfs 7.3G 0 7.3G 0% /dev/shm >>>> /dev/xvdb 37G 1.2G 34G 4% /mnt >>>> /dev/xvdc 37G 177M 35G 1% /mnt2 >>>> /dev/xvds 1000G 805G 195G 81% /vol0 >>>> >>>> C $df -i >>>> Filesystem Inodes IUsed IFree IUse% Mounted on >>>> /dev/xvda1 524288 97938 426350 19% / >>>> tmpfs 1906897 1 1906896 1% /dev/shm >>>> /dev/xvdb 2457600 54 2457546 1% /mnt >>>> /dev/xvdc 2457600 24 2457576 1% /mnt2 >>>> /dev/xvds 755218352 24024 755194328 1% /vol0 >>>> root@ip-10-204-136-223 ~]$ >>>> >>>> C $ df -h >>>> Filesystem Size Used Avail Use% Mounted on >>>> /dev/xvda1 7.9G 3.4G 4.5G 44% / >>>> tmpfs 7.3G 0 7.3G 0% /dev/shm >>>> /dev/xvdb 37G 1.2G 34G 4% /mnt >>>> /dev/xvdc 37G 177M 35G 1% /mnt2 >>>> /dev/xvds 1000G 820G 181G 82% /vol0 >>>> >>>> The devices may be ~80% full but that still leaves ~200G free on each. >>>> My spark-env.sh has >>>> >>>> export SPARK_LOCAL_DIRS="/vol0/spark" >>>> >>>> I have manually verified that on each slave the only temporary files >>>> are stored on /vol0, all looking something like this >>>> >>>> >>>> /vol0/spark/spark-f05d407c/spark-fca3e573/spark-78c06215/spark-4f0c4236/20/rdd_8_884 >>>> >>>> So it looks like all the files are being stored on the large drives >>>> (incidentally they're AWS EBS volumes, but that's the only way to get >>>> enough storage). My process crashed before with a slightly different >>>> exception under the same circumstances: kryo.KryoException: >>>> java.io.IOException: No space left on device >>>> >>>> These both happen after several hours and several GB of temporary files. >>>> >>>> Why does Spark think it's run out of space? >>>> >>>> TIA >>>> >>>> Joe >>>> >>>> Stack trace 1: >>>> >>>> org.apache.spark.shuffle.MetadataFetchFailedException: Missing an >>>> output location for shuffle 0 >>>> at >>>> org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$1.apply(MapOutputTracker.scala:384) >>>> at >>>> org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$1.apply(MapOutputTracker.scala:381) >>>> at >>>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) >>>> at >>>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) >>>> at >>>> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) >>>> at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108) >>>> at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) >>>> at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108) >>>> at >>>> org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:380) >>>> at >>>> org.apache.spark.MapOutputTracker.getServerStatuses(MapOutputTracker.scala:176) >>>> at >>>> org.apache.spark.shuffle.hash.BlockStoreShuffleFetcher$.fetch(BlockStoreShuffleFetcher.scala:42) >>>> at >>>> org.apache.spark.shuffle.hash.HashShuffleReader.read(HashShuffleReader.scala:40) >>>> at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:92) >>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:280) >>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:247) >>>> at >>>> org.apache.spark.rdd.CoalescedRDD$$anonfun$compute$1.apply(CoalescedRDD.scala:93) >>>> at >>>> org.apache.spark.rdd.CoalescedRDD$$anonfun$compute$1.apply(CoalescedRDD.scala:92) >>>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371) >>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) >>>> at >>>> org.apache.spark.serializer.SerializationStream.writeAll(Serializer.scala:109) >>>> at >>>> org.apache.spark.storage.BlockManager.dataSerializeStream(BlockManager.scala:1177) >>>> at org.apache.spark.storage.DiskStore.putIterator(DiskStore.scala:78) >>>> at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:787) >>>> at >>>> org.apache.spark.storage.BlockManager.putIterator(BlockManager.scala:638) >>>> at >>>> org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:145) >>>> at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:70) >>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:245) >>>> at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31) >>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:280) >>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:247) >>>> at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31) >>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:280) >>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:247) >>>> at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31) >>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:280) >>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:247) >>>> at >>>> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68) >>>> at >>>> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) >>>> at org.apache.spark.scheduler.Task.run(Task.scala:56) >>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:200) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> at java.lang.Thread.run(Thread.java:745) >>>> >>>> Stack trace 2: >>>> >>>> 15/02/22 02:47:08 WARN scheduler.TaskSetManager: Lost task 282.0 in >>>> stage 25.1 (TID 22644): com.esotericsoftware. >>>> kryo.KryoException: java.io.IOException: No space left on device >>>> at com.esotericsoftware.kryo.io.Output.flush(Output.java:157) >>>> at com.esotericsoftware.kryo.io.Output.require(Output.java:135) >>>> at >>>> com.esotericsoftware.kryo.io.Output.writeAscii_slow(Output.java:446) >>>> at >>>> com.esotericsoftware.kryo.io.Output.writeString(Output.java:306) >>>> at >>>> com.esotericsoftware.kryo.serializers.DefaultSerializers$StringSerializer.write(DefaultSerializers.java:153) >>>> at >>>> com.esotericsoftware.kryo.serializers.DefaultSerializers$StringSerializer.write(DefaultSerializers.java:146) >>>> at >>>> com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) >>>> at >>>> carbonite.serializer$print_collection.invoke(serializer.clj:41) >>>> at clojure.lang.Var.invoke(Var.java:423) >>>> at >>>> carbonite.ClojureCollSerializer.write(ClojureCollSerializer.java:19) >>>> at >>>> com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) >>>> at >>>> org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:130) >>>> at >>>> org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:195) >>>> at >>>> org.apache.spark.util.collection.ExternalSorter.spillToMergeableFile(ExternalSorter.scala:303) >>>> at >>>> org.apache.spark.util.collection.ExternalSorter.spill(ExternalSorter.scala:254) >>>> at >>>> org.apache.spark.util.collection.ExternalSorter.spill(ExternalSorter.scala:83) >>>> at >>>> org.apache.spark.util.collection.Spillable$class.maybeSpill(Spillable.scala:87) >>>> at >>>> org.apache.spark.util.collection.ExternalSorter.maybeSpill(ExternalSorter.scala:83) >>>> at >>>> org.apache.spark.util.collection.ExternalSorter.maybeSpillCollection(ExternalSorter.scala:237) >>>> at >>>> org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:206) >>>> at >>>> org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:56) >>>> at >>>> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68) >>>> at >>>> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) >>>> at org.apache.spark.scheduler.Task.run(Task.scala:56) >>>> at >>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:200) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> at java.lang.Thread.run(Thread.java:745) >>>> Caused by: java.io.IOException: No space left on device >>>> at java.io.FileOutputStream.writeBytes(Native Method) >>>> at java.io.FileOutputStream.write(FileOutputStream.java:345) >>>> at >>>> org.apache.spark.storage.DiskBlockObjectWriter$TimeTrackingOutputStream$$anonfun$write$3.apply$mcV$sp(BlockObjectWriter.scala:86) >>>> at org.apache.spark.storage.DiskBlockObjectWriter.org >>>> $apache$spark$storage$DiskBlockObjectWriter$$callWithTiming(BlockObjectWriter.scala:221) >>>> at >>>> org.apache.spark.storage.DiskBlockObjectWriter$TimeTrackingOutputStream.write(BlockObjectWriter.scala:86) >>>> at >>>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:122) >>>> at >>>> org.xerial.snappy.SnappyOutputStream.dumpOutput(SnappyOutputStream.java:300) >>>> at >>>> org.xerial.snappy.SnappyOutputStream.rawWrite(SnappyOutputStream.java:247) >>>> at >>>> org.xerial.snappy.SnappyOutputStream.write(SnappyOutputStream.java:107) >>>> >>>> >>>> >>> >>
