Vivek1106-04 commented on PR #57528:
URL: https://github.com/apache/spark/pull/57528#issuecomment-5155504764
@sunchao! Both fixed, The tie break now rotates: the window of blocks tied
at the cutoff that a map task records starts at mapTaskId mod numTied and
wraps, so a tied reducer is averaged away on roughly one mapper in numTied
instead of on all of them — cap unchanged, map status still deterministic,
avgSize unaffected since only which blocks are recorded changes, not how many.
Your example is the regression test at 2000 map statuses: it asserts each
status still records exactly 100 blocks and every one of the 101 tied reducers
sums to more than 98% of its real size, and without the rotation it fails at
20576000 was not greater than 2.1986692E8, your arithmetic exactly. On the
heap: I reproduced your measurement first — on a real MapOutputTrackerMaster
holding a 10000 map task stage, the retained map statuses are 6004 bytes per
map task, 57 MiB for the stage, matching your 5775 and 57,752,240, so the five
bytes in the PR body were writeExternal payload and I've corrected that c
laim. hugeBlockSizes is now two parallel primitive arrays sorted by reduce id
that getSizeForBlock binary searches, which brings the same measurement to 668
bytes per map task and 6 MiB for the stage, about 63 MiB at 100000 mappers
against your projected 551 MiB; the benchmark runs that measurement itself and
prints it. Since swapping a hash lookup for a binary search invites the obvious
question, I benchmarked reading every block size out of every map status too:
6.5 ns per lookup with the map versus 3.9 ns with the arrays at 50000
partitions, the reduce id no longer being boxed on every get. Two costs worth
naming: the serialized statuses for that stage grow from 22 KiB to 27 KiB
because each map task now records a different id set and compresses slightly
worse, and the rotation does nothing when blocks strictly above the cutoff
already fill the budget, since then no tie is recorded on any mapper — that is
the cap's semantics rather than the tie break's, so I left it alone, but
happy to revisit if you'd rather it were handled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
