We have extended the implementation of MR3 so that all partition
inputs can be fetched with a single call, e.g.:
rssShuffleClient.readPartition(..., 0, 100)
Now, Hive-MR3 with Celeborn runs as fast as Hive-MR3 with its own shuffle
handlers when tested with 10TB TPC-DS benchmark. For some queries, it is
even noticeably faster.
Thanks,
--- Sungwoo
On Thu, 13 Jul 2023, [email protected] wrote:
Hi Team,
I have a question on how a reducer should fetch the output of mappers.
As an example, consider this standard scenario:
1. There are 100 mapper and 50 reducers.
2. Each mapper creates 50 partitions, each of which is to be fetched by the
corresponding reducer.
3. Each reducer is responsible for a single partition and tries to fetch 100
partitions (one from each mapper).
In our current implementation, a reducer calls
rssShuffleClient.readPartition() 100 times (one for each mapper):
rssShuffleClient.readPartition(..., mapIndex, mapIndex + 1)
My question is: if reducers start after the completion of all mappers, can we
call (or should we try to call) rssShuffleClient.readPartition() only once,
as in?
rssShuffleClient.readPartition(..., 0, 100)
My understanding of remote shuffle service (like Magnet for Spark) is that
all the partitions destined to the same reducer are automatically merged by
the shuffle service, so we thought that just a single call might be enough.
Thanks,
--- Sungwoo Park
