Thanks, that's just what I needed to fix the problem! Looks like it's singing and dancing now ...
Jim -----Original Message----- From: Paul Brown [mailto:p...@mult.ifario.us] Sent: Thursday, March 06, 2014 1:50 PM To: dev@spark.apache.org Subject: Re: Spark Streaming and Storehaus -- example? I'd hazard that this is a generic issue. The "store" is in the context of the driver code, not the worker code, and that's why Spark is trying to send it off to a worker for execution. It's not serializable (and shouldn't be...), so that fails. Try making a Scala object that lives on the worker side (e.g., by static initialization) and then access it that way. — p...@mult.ifario.us | Multifarious, Inc. | http://mult.ifario.us/ On Thu, Mar 6, 2014 at 1:44 PM, Jim Donahue <jdona...@adobe.com> wrote: > After watching Ryan Weald's talk about integrating Spark Streaming > with Algebird and Storehaus, I decided that I had to give this a try! > :-) But I'm having some rather basic problems. > > My code looks a lot like the example Ryan gives. Here's the basic > structure: > > dstream.foreach(rdd => if(rdd.count != 0){ val data = > rdd.collect.toMap; store.multiMerge(data)}) > > where dstream is my Spark DStream and store is a Storehaus MySqlLongStore. > > The problem is that the Spark Streaming CheckpointWriter wants to > serialize the store object and it can't (I end up getting a > NotSerializableException). > > Anybody have an example of working code? Is this problem specific to > MySqlLongStores, or is this generic? > > > Thanks, > > Jim Donahue > Adobe >
