aah, yes. I missed that. I looked into the code. Both TreeBroadcast and HttpBroadcast don't do send or write respectively.. Will wait for other inputs.
On Tue, Nov 19, 2013 at 10:40 PM, Eugen Cepoi <[email protected]> wrote: > Yes sure for usual tests it is fine, but the broadcast is only done if we > are not in local mode (at least seems so). > > In SparkContext we have def broadcast[T](value: T) = > env.broadcastManager.newBroadcast[T](value, isLocal) > the is local is computed from the master name ("local" or "local[..."). > Now If we look int HttpBroadcast we see > if (!isLocal) { > HttpBroadcast.write(id, value_) > } > > The broadcast is not done in local. I guess this is an optimization in > case we run multiple threads sharing the same broadcasted variable. But > perhaps am I missing something? > > > 2013/11/19 Sriram Ramachandrasekaran <[email protected]> > >> Trying local[m], where m is the number of workers. For tests, local[2] >> should be ideal. This is the way to accomplish writing tests for Spark code >> generally. >> >> >> On Tue, Nov 19, 2013 at 10:03 PM, Eugen Cepoi <[email protected]>wrote: >> >>> Maybe a bug with HttpBroadcast or maybe my fault but can't find where :) >>> >>> The problem: >>> At the beginning a job computes a treemap(string, someobject) with a >>> custom order (some dummy lowercase), this treemap is broadcasted. >>> Then i use this map to do some matching against input rdd (excluding >>> those that don't exist). >>> What happens? In local (bc is in that case not used) or by passing all >>> the treemap without broadcast I got more than 3M matchings, after broadcast >>> it falls to 20K. >>> >>> Replacing HttpBroadcastFactory with TreeBroadcastFactory solves the >>> problem (I obtain expected results). I am trying to implement a test case >>> to reproduce it, but it is quite tricky in that case... >>> >>> BTW is there a way to reproduce the broadcast mechanism in local (I see >>> that the SparkEnv instance is shared as static, so I guess there is no easy >>> way)? >>> >>> Thanks, >>> Eugen >>> >> >> >> >> -- >> It's just about how deep your longing is! >> > > -- It's just about how deep your longing is!
