Hi, It should generate the same no of partitions as the no. of splits. Howd you check no of partitions.? Also please paste your file size and hdfs-site.xml and mapred-site.xml here.
Thanks and Regards, Archit Thakur. On Sat, Apr 18, 2015 at 6:20 PM, Wenlei Xie <wenlei....@gmail.com> wrote: > Hi, > > I am wondering the mechanism that determines the number of partitions > created by SparkContext.sequenceFile ? > > For example, although my file has only 4 splits, Spark would create 16 > partitions for it. Is it determined by the file size? Is there any way to > control it? (Looks like I can only tune minPartitions but not maxPartitions) > > Thank you! > > Best, > Wenlei > > >
