> At the time of the below reported problem there was a swarm
> of OOMEs. Three regionservers went down within minutes of each
> other.
> The load at the time was four reducers that were writing
> serialized Document objects back. I suspect the writes were
> all hitting the same few regions (<= 4). These writes were
> in addition to the crawler write load of ~100-200
> objects/second.
Statistics from one invocation of that job:
Counters: 13
File Systems
Local bytes read=6599658380
Local bytes written=11191638985
Job Counters
Launched reduce tasks=5
Launched map tasks=373
Map-Reduce Framework
Reduce input groups=61134
Combine output records=0
Map input records=781246
Reduce output records=61134
Map output bytes=4514592311
Map input bytes=0
Combine input records=0
Map output records=61134
Reduce input records=61134
The map output bytes is a good ballpark for the amount of
data that will collectively be committed back to HBase by
the reducers.
- Andy
