The following 2 could solve the issue to some extent. But these 2 are not automatically done by hadoop. User needs to set things before submitting the job. (1) Enabling map output compression using configuration property mapreduce.map.output.compress. (2) Use combiner so that possibly less amount of intermediate data will be emitted by mapper.
-Ravi On 1/3/11 5:58 PM, "Debbie Fu" <[email protected]> wrote: > Hi, > Is there any possibility that the intermediate output might be too large to > store it in the local disk? > If there is, what does hadoop do to solve the problem? > Thanks.
