Hello,

I assume you want to associate {a,b}, {c,d,e}, and {f} into sets.

One way to do this is by associating some value with each flag and then 
emitting the data associated with that value. For example,

flag
a
b
flag
c
d
e
flag
f

I define flag,a,b,c,d,e,f to be the key while in the Mapper context.

Whenever the mapper sees a key, it will emit <UID, Key>. UID is some unique 
identifier associated with a certain set, and Key is the key that was passed 
into the mapper. We are essentially inverting the association here.

Let's step through this testcase.
 1. Choose UID = mapper1flag1.
 2. <flag, null> -> Mapper -> <mapper1flag1, flag>
 3. We have reached a flag, so we change the UID = mapper1flag2.
 4. <a, null> -> Mapper -> <mapper1flag2, a>
 5. <b, null> -> Mapper -> <mapper1flag2, b>
 6. <flag, null> -> Mapper -> <mapper1flag2, flag>
 7. We have reached a flag, so we change the UID = mapper1flag3.
 8. <c, null> -> Mapper -> <mapper1flag3, c>
 9. <d, null> -> Mapper -> <mapper1flag3, d>
10. <e, null> -> Mapper -> <mapper1flag3, e>
11. <flag, null> -> Mapper -> <mapper1flag3, flag>
12. We have reached a flag, so we change the UID = mapper1flag4.
13. <f, null> -> Mapper -> <mapper1flag3, f>
14. EOF

Then the reducers will collect all values with the same UID, so here is what we 
get:

1. <mapper1flag1, {flag}> -> Reducer -> <{}, null>
2. <mapper1flag2, {a,b,flag}> -> Reducer -> <{a,b}, null>
3. <mapper1flag3, {c,d,e,flag}> -> Reducer -> <{c,d,e}, null>
4. <mapper1flag4, {f}> -> Reducer -> <{f}, null>

Hopefully this solves your problem.

On Sat, October 4, 2008 2:48 am, Zhou, Yunqing said:
> but the close() function doesn't supply me a Collector to put pairs in.
> 
> Is it reasonable for me to store a reference of the collector in advance?
> 
> 
> I'm not sure if the collector is still available then.
> 
> 
> 
> 
> On Sat, Oct 4, 2008 at 12:17 PM, Joman Chu <[EMAIL PROTECTED]> wrote:
> 
> 
>> Hello,
>> 
>> Does MapReduceBase.close() fit your needs? Take a look at 
>> http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/mapred
>> /MapReduceBase.html#close()
>> 
>> On Fri, October 3, 2008 11:36 pm, Zhou, Yunqing said:
>>> the input is as follows. flag a b flag c d e flag f
>>> 
>>> then I used a mapper to first store values and then emit them all
>>> when met with a line contains "flag" but when the file reached its
>>> end, I have no chance to emit the last record.(in this case ,f) so how
>>> can I detect
>> the
>>> mapper's end of its life , or how can I emit a last record before a
>> mapper
>>> exits.
>>> 
>>> Thanks
>>> 
>> 
>> Have a good one, -- Joman Chu Carnegie Mellon University School of Computer
>> Science 2011 AIM: ARcanUSNUMquam
>> 
>> 
> 


-- 
Joman Chu
Carnegie Mellon University
School of Computer Science 2011
AIM: ARcanUSNUMquam

Reply via email to