Thank you so much! I was trying for a singleton and opted against a class
but clearly this backfired. Clearly time to revisit Scala lessons. Thanks
again
On Mon, Jun 23, 2014 at 1:16 PM, Marcelo Vanzin <van...@cloudera.com> wrote:
> "object" in Scala is similar to a class with only static fields /
> methods in Java. So when you set its fields in the driver, the
> "object" does not get serialized and sent to the executors; they have
> their own copy of the class and its static fields, which haven't been
> initialized.
>
> Use a proper class, instantiate it, and then use it in the executors. e.g.
>
> class Foo extends Serializable {
> ...
> }
>
> val foo = new Foo()
> foo.field1 = "blah"
>
> lines.map(line => { println(foo) }) // now you should see the field
> values you set.
>
>
>
> On Mon, Jun 23, 2014 at 7:44 AM, Yana Kadiyska <yana.kadiy...@gmail.com>
> wrote:
> > Hi folks, hoping someone can explain to me what's going on:
> >
> > I have the following code, largely based on RecoverableNetworkWordCount
> > example
> > (
> https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/streaming/RecoverableNetworkWordCount.scala
> ):
> >
> > I am setting fields on an object that gets accessed within the map
> function.
> > But the workers do not see the set values. Could someone help me
> understand
> > what is going on? I suspect the serialization of the object happens not
> when
> > I think it does....
> >
> > object Foo{
> > var field1 = -1
> > var field2 = -2
> > }
> >
> > def main(args: Array[String]) {
> > //open a yaml file
> > Foo.field1 = value_from_yaml
> > Foo.field2 = value from_yaml
> >
> > val ssc = StreamingContext.getOrCreate(checkpointDirectory,
> > () => { createContext() })
> > ssc.start()
> > }
> >
> > def createContext(){
> > // create streaming context
> > println(Foo) <--
> > foo's fields are set
> > lines.map(line =>{
> > println(Foo) <--
> > foo's fields are not set
> > })
> > }
>
>
>
> --
> Marcelo
>
