Hi, This should work. How are you checking the no. of partitions.?
Thanks and Regards, Archit Thakur. On Mon, Apr 20, 2015 at 7:26 PM, mas <[email protected]> wrote: > Hi, > > I aim to do custom partitioning on a text file. I first convert it into > pairRDD and then try to use my custom partitioner. However, somehow it is > not working. My code snippet is given below. > > val file=sc.textFile(filePath) > val locLines=file.map(line => line.split("\t")).map(line=> > ((line(2).toDouble,line(3).toDouble),line(5).toLong)) > val ck=locLines.partitionBy(new HashPartitioner(50)) // new > CustomPartitioner(50) -- none of the way is working here. > > while reading the file using "textFile" method it automatically partitions > the file. However when i explicitly want to partition the new rdd > "locLines", It doesn't appear to do anything and even the number of > partitions are same which is created by sc.textFile(). > > Any help in this regard will be highly appreciated. > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Custom-Partitioning-Spark-tp22571.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
