Additionally, you can set mapred.local.dir to be a comma-separated list of paths that reside on multiple disks -- this spreads I/O plus gives you additional space.
But I suppose if a single Mapper is writing a huge amount of data for a single partition output, it may cause a disk fill-up. Please correct me if am wrong here. On Mon, Jan 3, 2011 at 5:58 PM, Debbie Fu <[email protected]> wrote: > Hi, > Is there any possibility that the intermediate output might be too large to > store it in the local disk? > If there is, what does hadoop do to solve the problem? > Thanks. > > -- > Best regards! > > -- Harsh J www.harshj.com
