Hi Jack, Are you writing out to disk? Or it sounds like Spark is spilling to disk (RAM filled up) and it's running out of disk space.
Cheers Andy On Thu, Sep 24, 2015 at 4:29 PM, Jack Yang <j...@uow.edu.au> wrote: > Hi folk, > > > > I have an issue of graphx. (spark: 1.4.0 + 4 machines + 4G memory + 4 CPU > cores) > > Basically, I load data using GraphLoader.edgeListFile mthod and then count > number of nodes using: graph.vertices.count() method. > > The problem is : > > > > *Lost task 11972.0 in stage 6.0 (TID 54585, 192.168.70.129): > java.io.IOException: No space left on device* > > * at java.io.FileOutputStream.writeBytes(Native Method)* > > * at java.io.FileOutputStream.write(FileOutputStream.java:345)* > > > > when I try a small amount of data, the code is working. So I guess the > error comes from the amount of data. > > This is how I submit the job: > > > > spark-submit --class "myclass" > > --master spark://hadoopmaster:7077 (I am using standalone) > > --executor-memory 2048M > > --driver-java-options "-XX:MaxPermSize=2G" > > --total-executor-cores 4 my.jar > > > > > > Any thoughts? > > Best regards, > > Jack > > > -- Andy Huang | Managing Consultant | Servian Pty Ltd | t: 02 9376 0700 | f: 02 9376 0730| m: 0433221979
