Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/19788#discussion_r152891792
--- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ---
@@ -812,10 +812,13 @@ private[spark] object MapOutputTracker extends
Logging {
logError(errorMessage)
throw new MetadataFetchFailedException(shuffleId, startPartition,
errorMessage)
} else {
+ var totalSize = 0L
for (part <- startPartition until endPartition) {
- splitsByAddress.getOrElseUpdate(status.location, ArrayBuffer())
+=
- ((ShuffleBlockId(shuffleId, mapId, part),
status.getSizeForBlock(part)))
+ totalSize += status.getSizeForBlock(part)
--- End diff --
This can be simplified like: `val totalSize = (startPartition until
endPartition).map(status.getSizeForXXX).sum`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
