Under certain conditions, Scala will pull an entire class instance into a closure instead of just particular value members of that class, which can knee-cap Spark's serialization of functions in multiple ways. I tripped over this last week, and wrote up the experience on my blog. I've since noticed a couple other user descriptions of the same scenario, so maybe the post will be interesting to the community:
http://erikerlandson.github.io/blog/2015/03/31/hygienic-closures-for-scala-function-serialization/ --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
