On Thu, Aug 12, 2010 at 11:56 AM, Stuart Smith <[email protected]> wrote: > What I actually ended up doing was catching the OOME's in my M/R tasks, and > looking at the cell size. One of the cells was 500 MB :|. So that was bad. > I've taken to avoiding large cells in the M/R task, and things have smoothed > out. > > It looks like I should just be a little more circumspect with how much data I > cram in a cell. Mostly I limit them to 64 MB, but for one particular tasks I > limited to 512 MB.. and I'm getting a decent amount of data now, so > inevitably I hit the limit... >
Good one. If your cells are that size, you might be better off using hdfs direct perhaps keeping index to data up in hbase? Or, can you break up the cell content? St.Ack
