The parser does not need to be serializable. In the line: lines.map(line => JSONParser.parse(line))
... the parser is called but there is no parser object that with state that can be serialized. Are you sure it does not work? The error message alluded to originally refers to an object not shown in the code, so I'm not 100% sure this was the original issue. If you want, the equivalent of "setup()" is really "writing some code at the start of a call to mapPartitions()" On Mon, Aug 4, 2014 at 8:40 AM, Fengyun RAO <raofeng...@gmail.com> wrote: > Thanks, Ron. > > The problem is that the "parser" is written in another package which is not > serializable. > > In mapreduce, I could create the "parser" in the map setup() method. > > Now in spark, I want to create it for each worker, and share it among all > the tasks on the same work node. > > I know different workers run on different machine, but it doesn't have to > communicate between workers. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
