Well, for the record, what I ended up doing is to use DistributedCache
(http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/filecache/DistributedCache.html)
to locally cache the reference file at every Mapper/Reducer and go
through all reference records for each record provided by the
TaskTracker.

Ian.

On Tue, Aug 5, 2008 at 19:48, Theocharis Ian Athanasakis
<[EMAIL PROTECTED]> wrote:
> Apologies for misphrasing my question.
>
> Let me rephrase it: Using the Hadoop Java APIs is there a suggested
> way of doing a pair-wise comparison between all LineRecords in a file?
>
> More generically: is there a Hadoop Java API design pattern for a
> reducer to iterate through all the records in another file stored on
> HDFS?
>
> I'm currently using the DistributeCache class to cache the reference
> file locally. The shard a reducer is examining is always a part of the
> reference file. My reducer, then, ends up doing all the comparisons
> between its shard and the reference file.
>
> When all of these get combined, I have my pair-wise comparison between
> all records.
>
> Any better ways?
>
> On Tue, Aug 5, 2008 at 11:20, Theocharis Ian Athanasakis
> <[EMAIL PROTECTED]> wrote:
>> What's the proposed the design pattern for a reducer that needs two
>> sets of inputs?
>> Are there any source code examples?
>>
>> Thanks :)
>>
>

Reply via email to