On Wed, May 5, 2010 at 8:54 PM, Ted Dunning <ted.dunn...@gmail.com> wrote: > One big win would be to use Benson's collections instead of a normal map > (you may already have this).
Of course, already doing so ! > Another problem is the expectation that a map-reduce should be happy with > 500MB of memory. I don't think that is a universally reasonable Agree with this sentiment; I might pick a somewhat higher value as the reasonable max memory for a worker... but I came from Google land where RAM grows on trees. See below for more on this. > common than others. Regardless, however, it is reasonable to expect that if > you see about the same amount of data that you will see about the same > number of items. In a reasonably scaling system, the amount of data each > node in the cluster sees will be roughly constant as the input and cluster > size scale together. In a map-reduce architecture, you even get this > property if the input size and number of mappers and reducers scale > together. ... which is the assumption that made me guess this lookup table isn't going to get big. But if I'm right, it doesn't hurt to cap the size of the lookup. Which I can commit. But maybe that's not the issue either. > One observation that makes me suspect that the problem is not the size of > this table that Sean has mentioned is the fact that Tamas has observed that > he doesn't get more tasks when he scales the cluster size. That may be that > Hadoop is happy with the number of splits and has to be forced to split more > (by setting max split size or number of mappers). The problem might also > be that this program as designed just needs 3GB to complete and efforts to > force it into 500MB will always fail. Tamas, are you configuring the number of mappers and reducers with command line args? It will not by default run many workers. I don't know of any stage of this computation that should require significant memory, so it just can't need this much memory. Besides this little lookup map, the worst thing it does is load a whole row of the co-occurrence matrix in memory. Actually I think I know the answer, perhaps. This is one of two stages that relies heavily on the combiner for efficiency. It sets io.sort.mb (the size of the buffer used to hold map output records before combining and spilling to disk) high. I suspect the issue is here since it seems to fail when allocating space for the map output somehow. Its value is 1000MB, or, if you've set a heap size via mapred.child.java.opts, half of the heap size you allocate. It actually does this via a crude regex looking for "Xmx[0-9]+m" in the string. So I can imagine that if you don't set it, this buffer overruns the heap. Or if you do set it, low, then other memory usage overruns half the heap and it fails again at some point. You are welcome to test this by modifying setIOSort() in RecommenderJob. Just crank down that "1000", or divide heapMB by a larger value. This should let you run with somewhat smaller heap sizes. If that's the culprit I'm happy to turn down these values a bit.
