Dear Hama developers,
I still have a problem setting the split size of an HDFS input file
using Hama 0.6.4. For example, when I use:
BSPJob job = new BSPJob(conf,BSPop.class);
job.setNumBspTask(10);
job.setLong("bsp.min.split.size",10000L); // 10000 bytes
For a small file with 2 blocks, this will use only 2 BSP tasks (one for
each block), instead of 10.
This used to work in Hama 0.5.0.
Any suggestions?
Thanks.
Leonidas Fegaras
On 01/04/2013 05:45 PM, Edward J. Yoon wrote:
Hello,
than a block. But if you have more nodes in your cluster than data blocks,
you may get faster execution if you allow splits smaller than a block. Is
You're right. So, we're working on partitioning issues now.
you may get faster execution if you allow splits smaller than a block. Is
there any way to use splits smaller than a block in Hama 0.6.0?
Yes. But, Hama 0.6.1 version will support it.
On Sat, Jan 5, 2013 at 4:59 AM, Leonidas Fegaras <[email protected]> wrote:
Dear Hama developers,
It seems that the splits generated by the FileInputFormat in Hama 0.6.0
cannot be smaller than a block. In Hama 0.5.0, I could set any split size
using job.set("bsp.min.split.size",...) and set the task numbers using
job.setNumBspTask(...). This is ignored by Hama 0.6.0 for a split smaller
than a block. But if you have more nodes in your cluster than data blocks,
you may get faster execution if you allow splits smaller than a block. Is
there any way to use splits smaller than a block in Hama 0.6.0?
Thanks for your help,
Leonidas
