Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/2030#discussion_r16435754
--- Diff:
core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ---
@@ -109,99 +155,30 @@ private[spark] class TorrentBroadcast[T: ClassTag](
private def readObject(in: ObjectInputStream) {
in.defaultReadObject()
TorrentBroadcast.synchronized {
--- End diff --
block size is limited to 4mb iirc - while deserializing two 100mb
broadcasts (as mentioned in usecase) in parallel will mean first has to
complete before second (or any other) can even start : so 50 requests have to
finish (half of which are pulling 4 mb each).
Even assuming both cores are pulling same broadcast, assuming reasonable
shuffle and large enough sizes, they could complement each other (best case,
reduce time taken by half, worst case, double IO assuming horrible shuffle or
single block).
From latency point of view, current option is not very good when waves of
tasks are getting scheduled and we have multiple cores per executors : from
network IO incurred point of view, maybe it is better.
Tradeoff question is, which is dominating.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
