Re: Is it common in spark to broadcast a 10 gb variable?

[Guillaume Pitel]

From my experience, it shouldn't be a problem since 0.8.1 (before that, the akka FrameSize was a limit). I've broadcast arrays of max 1.4GB so far Keep in mind that it will be stored in spark.local.dir so you must have room on the disk. Guillaume Hi, I asked a similar question a while ago, didn't get any answers. I'd like to share a 10 gb double array between 50 to 100 workers. The physical memory of workers is over 40 gb, so it can fit in each memory. The reason I'm sharing this array is that a cartesian operation is applied to this array, and I want to avoid network shuffling. 1. Is Spark broadcast built for pushing variables of gb size? Does it need special configurations (eg akka config, etc) to work under this condition? 2. (Not directly related to spark) Is the an upper limit for scala/java arrays other than the physical memory? Do they stop working when the array elements count exceeds a certain number? -- Guillaume PITEL, Président +33(0)6 25 48 86 80 eXenSa S.A.S. 41, rue Périer - 92120 Montrouge - FRANCE Tel +33(0)1 84 16 36 77 / Fax +33(0)9 72 28 37 05