On Apr 14, 2009, at 11:10 AM, Saptarshi Guha wrote:
Thanks. I am using 0.19, and to confirm, the map and combiner (in the map jvm) are run in *different* threads at the same time?
And the change was actually made in 0.18. So since then, the combiner is called 0, 1, or many times on each key in both the mapper and the reducer. It is called in a separate thread from the base application in the map (in the reduce task, the combiner is only use during the shuffle).
My native library is not thread safe, so I would have to implement locks. Aaron's email gave me hope(since the map and combiner would then be running sequentially), but this appears to make things complicated.
Yes, you'll probably need locks around your code that isn't thread safe. -- Owen
