On Mon, Feb 23, 2009 at 12:06 PM, Jimmy Wan <[email protected]> wrote:

> part of my map/reduce process could be greatly sped up by mapping
> key/value pairs in batches instead of mapping them one by one.
> Can I safely hang onto my OutputCollector and Reporter from calls to map?


Yes. You can even use them in the close, so that you can process the last
batch of records. *smile* One problem that you will quickly hit is that
Hadoop reuses the objects that are passed to map and reduce. So, you'll need
to clone them before putting them into the collection.

I'm currently running Hadoop 0.17.2.1. Is this something I could do in
> Hadoop 0.19.X?
>

I don't think any of this changed between 0.17 and 0.19, other than in 0.17
the reduce's inputs were always new objects. In 0.18 and after, the reduce's
inputs are reused.

-- Owen

Reply via email to