RangePartitioner does not actually provide a guarantee that all partitions
will be equal sized (that is hard), and instead uses sampling to
approximate equal buckets. Thus, it is possible that a bucket is left empty.
If you want the specified behavior, you should define your own partitioner.
It would look something like this (untested):
class AlphabetPartitioner extends Partitioner {
def numPartitions = 26
def getPartition(key: Any): Int = key match {
case s: String = s(0).toUpper - 'A'
}
override def equals(other: Any): Boolean =
other.isInstanceOf[AlphabetPartitioner]
override def hashCode: Int = 0
}
On Tue, Feb 17, 2015 at 7:05 PM, java8964 java8...@hotmail.com wrote:
Hi, Sparkers:
I just happened to search in google for something related to the
RangePartitioner of spark, and found an old thread in this email list as
here:
http://apache-spark-user-list.1001560.n3.nabble.com/RDD-and-Partition-td991.html
I followed the code example mentioned in that email thread as following:
scala import org.apache.spark.RangePartitioner
import org.apache.spark.RangePartitioner
scala val rdd = sc.parallelize(List(apple, Ball, cat, dog,
Elephant, fox, gas, horse, index, jet, kitsch, long,
moon, Neptune, ooze, Pen, quiet, rose, sun, talk,
umbrella, voice, Walrus, xeon, Yam, zebra))
rdd: org.apache.spark.rdd.RDD[String] = ParallelCollectionRDD[0] at
parallelize at console:13
scala rdd.keyBy(s = s(0).toUpper)
res0: org.apache.spark.rdd.RDD[(Char, String)] = MappedRDD[1] at keyBy at
console:16
scala res0.partitionBy(new RangePartitioner[Char, String](26,
res0)).values
res1: org.apache.spark.rdd.RDD[String] = MappedRDD[5] at values at
console:18
scala res1.mapPartitionsWithIndex((idx, itr) = itr.map(s = (idx,
s))).collect.foreach(println)
The above example is clear for me to understand the meaning of the
RangePartitioner, but to my surprise, I got the following result:
*(0,apple)*
*(0,Ball)*
(1,cat)
(2,dog)
(3,Elephant)
(4,fox)
(5,gas)
(6,horse)
(7,index)
(8,jet)
(9,kitsch)
(10,long)
(11,moon)
(12,Neptune)
(13,ooze)
(14,Pen)
(15,quiet)
(16,rose)
(17,sun)
(18,talk)
(19,umbrella)
(20,voice)
(21,Walrus)
(22,xeon)
(23,Yam)
(24,zebra)
instead of a perfect range index from 0 to 25 in old email thread. Why is
that? Is this a bug, or some new feature I don't understand?
BTW, the above environment I tested is in Spark 1.2.1 with Hadoop 2.4
binary release.
Thanks
Yong
