Github user gczsjdy commented on a diff in the pull request:
https://github.com/apache/spark/pull/19788#discussion_r152193203
--- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ---
@@ -812,10 +812,14 @@ private[spark] object MapOutputTracker extends
Logging {
logError(errorMessage)
throw new MetadataFetchFailedException(shuffleId, startPartition,
errorMessage)
} else {
+ var n = 0
+ var totalSize = 0L
for (part <- startPartition until endPartition) {
- splitsByAddress.getOrElseUpdate(status.location, ArrayBuffer())
+=
- ((ShuffleBlockId(shuffleId, mapId, part),
status.getSizeForBlock(part)))
+ n += 1
+ totalSize += status.getSizeForBlock(part)
}
--- End diff --
`n` can be `numPartitions`, and directly get by `endPartition -
startPartition` ?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
