If the file is big enough, you can try MLUtils.loadLibSVMFile with a minPartitions argument. This doesn't shuffle data but it might not give you the exact number of partitions. If you want to have the exact number, use RDD.repartition, which requires data shuffling. -Xiangrui
On Sun, Aug 10, 2014 at 9:35 AM, ayandas84 <[email protected]> wrote: > Hi, > > I am using spark-scala system to train distributed svm. For training svm I > am using the files in LIBSVM format. I want to partition a file into fixed > number of partititions, with each partition having equal number > of datapoints(assume that the number of datapoints in the file is exactly > divisible by number of partitions). How could I do that using spark-scala. > > Please help. Thanks. > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Partitioning-a-libsvm-format-file-tp11852.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
