From my experience, it shouldn't be a
problem since 0.8.1 (before that, the akka FrameSize was a limit).
I've broadcast arrays of max 1.4GB so far Keep in mind that it will be stored in spark.local.dir so you must have room on the disk. Guillaume
--
|
- Is it common in spark to broadcast a 10 gb variable? Aureliano Buendia
- Re: Is it common in spark to broadcast a 10 gb vari... Guillaume Pitel
- Re: Is it common in spark to broadcast a 10 gb vari... Josh Marcus
- Re: Is it common in spark to broadcast a 10 gb ... Stephen Boesch
- Re: Is it common in spark to broadcast a 10... Aureliano Buendia
- Re: Is it common in spark to broadcast ... Matei Zaharia
- Re: Is it common in spark to broad... Ryan Compton
- Re: Is it common in spark to b... Aureliano Buendia
- Re: Is it common in spark ... Ryan Compton