Glenn K. Lockwood created SPARK-1690:
----------------------------------------
Summary: RDD.saveAsTextFile throws scala.MatchError if RDD
contains empty elements
Key: SPARK-1690
URL: https://issues.apache.org/jira/browse/SPARK-1690
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 0.9.0
Environment: Linux/CentOS6, Spark 0.9.1, standalone mode against HDFS
from Hadoop 1.2.1
Reporter: Glenn K. Lockwood
Priority: Minor
The following pyspark code fails with a scala.MatchError exception if
sample.txt contains any empty lines:
file = sc.textFile('hdfs://gcn-3-45.ibnet0:54310/user/glock/sample.txt')
file.saveAsTextFile('hdfs://gcn-3-45.ibnet0:54310/user/glock/sample.out')
The resulting stack trace:
14/04/30 17:02:46 WARN scheduler.TaskSetManager: Lost TID 0 (task 0.0:0)
14/04/30 17:02:46 WARN scheduler.TaskSetManager: Loss was due to
scala.MatchError
scala.MatchError: 0 (of class java.lang.Integer)
at
org.apache.spark.api.python.PythonRDD$$anon$1.read(PythonRDD.scala:129)
at
org.apache.spark.api.python.PythonRDD$$anon$1.next(PythonRDD.scala:119)
at
org.apache.spark.api.python.PythonRDD$$anon$1.next(PythonRDD.scala:112)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at
org.apache.spark.rdd.PairRDDFunctions.org$apache$spark$rdd$PairRDDFunctions$$writeToFile$1(PairRDDFunctions.scala:732)
at
org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$2.apply(PairRDDFunctions.scala:741)
at
org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$2.apply(PairRDDFunctions.scala:741)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:109)
at org.apache.spark.scheduler.Task.run(Task.scala:53)
at
org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$1.apply$mcV$sp(Executor.scala:211)
at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:42)
at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:41)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at
org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:41)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:176)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
This can be reproduced with a sample.txt containing
"""
foo
bar
"""
and disappears if sample.txt is
"""
foo
bar
"""
--
This message was sent by Atlassian JIRA
(v6.2#6252)
