It's there in the error: No space left on device You ran out of disk space (local disk) on one of your machines.
On Mon, Mar 8, 2021 at 2:02 AM Sachit Murarka <connectsac...@gmail.com> wrote: > Hi All, > > I am getting the following error in my spark job. > > Can someone please have a look ? > > org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 > in stage 41.0 failed 4 times, most recent failure: Lost task 0.3 in stage > 41.0 (TID 80817, executor 193): com.esotericsoftware.kryo.KryoException: > java.io.IOException: No space left on device\n\tat > com.esotericsoftware.kryo.io.Output.flush(Output.java:188)\n\tat > com.esotericsoftware.kryo.io.Output.require(Output.java:164)\n\tat > com.esotericsoftware.kryo.io.Output.writeBytes(Output.java:251)\n\tat > com.esotericsoftware.kryo.io.Output.writeBytes(Output.java:237)\n\tat > com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ByteArraySerializer.write(DefaultArraySerializers.java:49)\n\tat > com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ByteArraySerializer.write(DefaultArraySerializers.java:38)\n\tat > com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651)\n\tat > org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:245)\n\tat > org.apache.spark.serializer.SerializationStream.writeValue(Serializer.scala:134)\n\tat > org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:241)\n\tat > org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:151)\n\tat > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)\n\tat > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)\n\tat > org.apache.spark.scheduler.Task.run(Task.scala:123)\n\tat > org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)\n\tat > org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)\n\tat > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)\n\tat > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat > java.lang.Thread.run(Thread.java:748)\nCaused by: java.io.IOException: No > space left on device\n\tat java.io.FileOutputStream.writeBytes(Native > Method)\n\tat > java.io.FileOutputStream.write(FileOutputStream.java:326)\n\tat > org.apache.spark.storage.TimeTrackingOutputStream.write(TimeTrackingOutputStream.java:58)\n\tat > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)\n\tat > java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)\n\tat > net.jpountz.lz4.LZ4BlockOutputStream.flush(LZ4BlockOutputStream.java:240)\n\tat > com.esotericsoftware.kryo.io.Output.flush(Output.java:186)\n\t... 19 > more\n\nDriver stacktrace:\n\tat > org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1889)\n\tat > org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1877)\n\tat > org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1876)\n\tat > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)\n\tat > scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)\n\tat > org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1876)\n\tat > org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)\n\tat > org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)\n\tat > scala.Option.foreach(Option.scala:257)\n\tat > org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:926)\n\tat > org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2110)\n\tat > org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2059)\n\tat > org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2048)\n\tat > org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)\n\tat > org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:737)\n\tat > org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)\n\tat > org.apache.spark.SparkContext.runJob(SparkContext.scala:2082)\n\tat > org.apache.spark.SparkContext.runJob(SparkContext.scala:2101)\n\tat > org.apache.spark.SparkContext.runJob(SparkContext.scala:2126)\n\tat > org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:945)\n\tat > org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)\n\tat > org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)\n\tat > org.apache.spark.rdd.RDD.withScope(RDD.scala:363)\n\tat > org.apache.spark.rdd.RDD.collect(RDD.scala:944)\n\tat > org.apache.spark.api.python.PythonRDD$.collectAndServe(PythonRDD.scala:166)\n\tat > org.apache.spark.api.python.PythonRDD.collectAndServe(PythonRDD.scala)\n\tat > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat > java.lang.reflect.Method.invoke(Method.java:498)\n\tat > py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)\n\tat > py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)\n\tat > py4j.Gateway.invoke(Gateway.java:282)\n\tat > py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)\n\tat > py4j.commands.CallCommand.execute(CallCommand.java:79)\n\tat > py4j.GatewayConnection.run(GatewayConnection.java:238)\n\tat > java.lang.Thread.run(Thread.java:748)\nCaused by: > com.esotericsoftware.kryo.KryoException: java.io.IOException: No space left > on device\n\tat > com.esotericsoftware.kryo.io.Output.flush(Output.java:188)\n\tat > com.esotericsoftware.kryo.io.Output.require(Output.java:164)\n\tat > com.esotericsoftware.kryo.io.Output.writeBytes(Output.java:251)\n\tat > com.esotericsoftware.kryo.io.Output.writeBytes(Output.java:237)\n\tat > com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ByteArraySerializer.write(DefaultArraySerializers.java:49)\n\tat > com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ByteArraySerializer.write(DefaultArraySerializers.java:38)\n\tat > com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651)\n\tat > org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:245)\n\tat > org.apache.spark.serializer.SerializationStream.writeValue(Serializer.scala:134)\n\tat > org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:241)\n\tat > org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:151)\n\tat > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)\n\tat > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)\n\tat > org.apache.spark.scheduler.Task.run(Task.scala:123)\n\tat > org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)\n\tat > org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)\n\tat > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)\n\tat > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\t... > 1 more\nCaused by: java.io.IOException: No space left on device\n\tat > java.io.FileOutputStream.writeBytes(Native Method)\n\tat > java.io.FileOutputStream.write(FileOutputStream.java:326)\n\tat > org.apache.spark.storage.TimeTrackingOutputStream.write(TimeTrackingOutputStream.java:58)\n\tat > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)\n\tat > java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)\n\tat > net.jpountz.lz4.LZ4BlockOutputStream.flush(LZ4BlockOutputStream.java:240)\n\tat > com.esotericsoftware.kryo.io.Output.flush(Output.java:186)\n\t... 19 > more\n\n"} > > > Kind Regards, > Sachit Murarka >
