Hello Owen, > > Could you perhaps elaborate on that 100 MB limit ? Is that due to a > > limit that is caused by the Java VM heap size ? If so, could that, > > for example, be increased to 512MB by setting mapred.child.java.opts > > to '-Xmx512m' ? > > A couple of points: > 1. The 100MB was just for ballpark calculations. Of course if you > have a large heap, you can fit larger values. Don't forget that the > framework is allocating big chunks of the heap for its own buffers, > when figuring out how big to make your heaps. > 2. Having large keys is much harder than large values. When doing a > N-way merge, the framework has N+1 keys and 1 value in memory at a > time.
Ok, that makes sense. Thanks for the information! Regards, Leon Mergen
