liupc commented on issue #23580: [SPARK-26660]Add warning logs for large taskBinary size URL: https://github.com/apache/spark/pull/23580#issuecomment-455963772 The Logs looks like below after changes: Driver: ``` val c = (0 to 1000000).map(_ => r.nextLong().toString).toSeq c: scala.collection.immutable.Seq[String] = Vector(-8824816803907433122, 676576201675227827, -4450800462103535332, 4941742638328912331, -629826214489158275, 3599102463125026709, -1565579140513476623, -4953176692696234676, -7337015974401685398, 6861152894389600367, 4204365614405597947, -5631388062093635828, 952349730665646011, -1737760484135545791, -3818941051984249617, 4577662835073288447, -1043227373757634713, -2626518287151915759, -8712872867573643992, -7649246031751633828, 1964631192671370827, -720939716164610968, -1113194824007301084, -5476000723857254882, 8672676133212526245, 8119817434619293651, -7264818629918867452, -2796934762986648699, -8077774542345832804, -7249714536899879581, 7106267389367680272, 1322500019139194514, 5502357868702972085, -6484503789... scala> sc.parallelize(Seq(1)).map(_ => c.length).collect() 19/01/21 14:18:09 WARN scheduler.DAGScheduler: Broadcasting large task binary with size 21.1 MiB res1: Array[Int] = Array(1000001) ``` Executor: ``` 19/01/21 14:18:09 INFO broadcast.TorrentBroadcast: Started reading broadcast variable 1 with 5 pieces (estimated total size 20.0 MiB) 19/01/21 14:18:09 INFO memory.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 4.0 MiB, free 260.0 MiB) 19/01/21 14:18:09 INFO memory.MemoryStore: Block broadcast_1_piece1 stored as bytes in memory (estimated size 4.0 MiB, free 256.0 MiB) 19/01/21 14:18:09 INFO memory.MemoryStore: Block broadcast_1_piece3 stored as bytes in memory (estimated size 4.0 MiB, free 252.0 MiB) 19/01/21 14:18:09 INFO memory.MemoryStore: Block broadcast_1_piece4 stored as bytes in memory (estimated size 920.1 KiB, free 251.1 MiB) 19/01/21 14:18:09 INFO client.TransportClientFactory: Successfully created connection to /10.132.4.49:36947 after 1 ms (0 ms spent in bootstraps) 19/01/21 14:18:09 INFO memory.MemoryStore: Block broadcast_1_piece2 stored as bytes in memory (estimated size 4.0 MiB, free 247.1 MiB) 19/01/21 14:18:09 INFO broadcast.TorrentBroadcast: Reading broadcast variable 1 took 353 ms 19/01/21 14:18:09 INFO memory.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 21.7 MiB, free 225.4 MiB) ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
