Pedro Rodriguez created SPARK-5385:
--------------------------------------
Summary: Calling textFile, parallelize, zip, then partitions
causes failure on some local[*]
Key: SPARK-5385
URL: https://issues.apache.org/jira/browse/SPARK-5385
Project: Spark
Issue Type: Bug
Reporter: Pedro Rodriguez
There is a bug in Spark core which produces the exception: "Can't zip RDDs with
unequal numbers of partitions"
General Steps to reproduce:
1. Run sc.textFiles
2. Run sc.parallelize
3. Zip results of top two
4. Call partitions on result of zip
5. Run for local, local[2], local[3],...
6. My machine (macbook air) fails on local[3].
Github repository with code example: https://github.com/EntilZha/spark-zip-bug
Steps to run: execute "sbt run", wait for failure
Stack trace:
java.lang.IllegalArgumentException: Can't zip RDDs with unequal numbers of
partitions
at
org.apache.spark.rdd.ZippedPartitionsBaseRDD.getPartitions(ZippedPartitionsRDD.scala:57)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:203)
at App$.main(App.scala:33)
at App.main(App.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
I am looking into the relevant classes, but insight would be appreciated. This
ticket may also be related to https://issues.apache.org/jira/browse/SPARK-2823
and https://issues.apache.org/jira/browse/SPARK-5351
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
