In case HttpBroadcast is used, spark creates a jetty server and uses http protocol for trasporting the broadcast variables to all workers. To do so it write the serialized broadcast variable into their corresponding files, and file names denote the broadcast id of the variable.
So broadcast_0 is the broadcast variable id 0's serialzed content. To test this, you can create a boradcast varibakle, print its id, then go to the spark.local.dir location to find the file and open it. Even though its a binary file you can find your object class name etc. On Mon, Apr 28, 2014 at 3:45 PM, wxhsdp <wxh...@gmail.com> wrote: > Hi, guys > > when i read in a file on spark shell, the console shows that broadcast_0 is > stored to memory. i guess it's > related to the file, but broadcast_0 is not the file itself, because they > have different size. what does broadcast_0 stand for? > > logs: > 14/04/28 18:02:50 INFO MemoryStore: ensureFreeSpace(138811) called with > curMem=138763, maxMem=311387750 > 14/04/28 18:02:50 INFO MemoryStore: Block broadcast_0 stored as values to > memory (estimated size 135.6 KB, free 296.7 MB) > a: org.apache.spark.rdd.RDD[String] = MappedRDD[4] at textFile at > <console>:12 > > also when i do actions on an transfomed RDD after shuffles, the console > shows: > 14/04/28 16:36:15.106 INFO CoarseGrainedExecutorBackend: Got assigned task > 56 > 14/04/28 16:36:15.106 INFO Executor: Running task ID 56 > 14/04/28 16:36:15.123 INFO BlockManager: Found block broadcast_0 > locally.... > > i think after shuffles, actions have no relation to the original RDD, why > the console shows: Found block broadcast_0 locally.... > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/what-does-broadcast-0-stand-for-tp4934.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > -- Sourav Chandra Senior Software Engineer · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · sourav.chan...@livestream.com o: +91 80 4121 8723 m: +91 988 699 3746 skype: sourav.chandra Livestream "Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd Block, Koramangala Industrial Area, Bangalore 560034 www.livestream.com
