Hi Jia, If you want to change the Tachyon block size, you can set the tachyon.user.block.size.bytes.default parameter ( http://tachyon-project.org/documentation/Configuration-Settings.html). You can set it via extraJavaOptions per job, or adding it to tachyon-site.properties.
I hope that helps, Gene On Mon, Jan 25, 2016 at 8:13 PM, Jia Zou <jacqueline...@gmail.com> wrote: > Dear all, > > First to update that the local file system data partition size can be > tuned by: > sc.hadoopConfiguration().setLong("fs.local.block.size", blocksize) > > However, I also need to tune Spark data partition size for input data that > is stored in Tachyon (default is 512MB), but above method can't work for > Tachyon data. > > Do you have any suggestions? Thanks very much! > > Best Regards, > Jia > > > ---------- Forwarded message ---------- > From: Jia Zou <jacqueline...@gmail.com> > Date: Thu, Jan 21, 2016 at 10:05 PM > Subject: Spark partition size tuning > To: "user @spark" <user@spark.apache.org> > > > Dear all! > > When using Spark to read from local file system, the default partition > size is 32MB, how can I increase the partition size to 128MB, to reduce the > number of tasks? > > Thank you very much! > > Best Regards, > Jia > >