[ https://issues.apache.org/jira/browse/SPARK-13290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen closed SPARK-13290. ----------------------------- > wholeTextFile and binaryFiles are really slow > --------------------------------------------- > > Key: SPARK-13290 > URL: https://issues.apache.org/jira/browse/SPARK-13290 > Project: Spark > Issue Type: Bug > Components: PySpark, Spark Core > Affects Versions: 1.6.0 > Environment: Linux stand-alone > Reporter: mathieu longtin > > Reading biggish files (175MB) with wholeTextFile or binaryFiles is extremely > slow. It takes 3 minutes in Java versus 2.5 seconds in Python. > The java process balloons to 4.3GB of memory and uses 100% CPU the whole > time. I suspects Spark reads it in small chunks and assembles it at the end, > hence the large amount of CPU. > {code} > In [49]: rdd = sc.binaryFiles(pathToOneFile) > In [50]: %time path, text = rdd.first() > CPU times: user 1.91 s, sys: 1.13 s, total: 3.04 s > Wall time: 3min 32s > In [51]: len(text) > Out[51]: 191376122 > In [52]: %time text = open(pathToOneFile).read() > CPU times: user 8 ms, sys: 691 ms, total: 699 ms > Wall time: 2.43 s > In [53]: len(text) > Out[53]: 191376122 > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org