Hi Team,
We are currently implementing a Celeborn client for our application
(called MR3 which is similar to Tez), and have a question on the internals
of Celeborn.
The question is whether a reducer should wait until the completion of all
mappers before starting to fetch mapper output. From the Celeborn API, it
seems like there is no need to wait until the completion of all mappers.
In other words, after a certain mapper finishes writing all its output, a
reducer can fetch the corresponding output from the mapper, regardless of
the status of other mappers.
On the other hand, we suspect that trying to fetch the output of a mapper
before the completion of other mappers occasionally triggers Premature EOF
Exception.
Any comment on this problem will be appreciated very much.
Thanks,
--- Sungwoo Park
