Ok, that's fair enough. But why things work up to the collect?during map and filter objects are not serialized? On Apr 15, 2014 12:31 AM, "Eugen Cepoi" <cepoi.eu...@gmail.com> wrote:
> Sure. As you have pointed, those classes don't implement Serializable and > Spark uses by default java serialization (when you do collect the data from > the workers will be serialized, "collected" by the driver and then > deserialized on the driver side). Kryo (as most other decent serialization > libs) doesn't require you to implement Serializable. > > For the missing attributes it's due to the fact that java serialization > does not ser/deser attributes from classes that don't impl. Serializable > (in your case the parent classes). > > > 2014-04-14 23:17 GMT+02:00 Flavio Pompermaier <pomperma...@okkam.it>: > >> Thanks Eugen for tgee reply. Could you explain me why I have the >> problem?Why my serialization doesn't work? >> On Apr 14, 2014 6:40 PM, "Eugen Cepoi" <cepoi.eu...@gmail.com> wrote: >> >>> Hi, >>> >>> as a easy workaround you can enable Kryo serialization >>> http://spark.apache.org/docs/latest/configuration.html >>> >>> Eugen >>> >>> >>> 2014-04-14 18:21 GMT+02:00 Flavio Pompermaier <pomperma...@okkam.it>: >>> >>>> Hi to all, >>>> >>>> in my application I read objects that are not serializable because I >>>> cannot modify the sources. >>>> So I tried to do a workaround creating a dummy class that extends the >>>> unmodifiable one but implements serializable. >>>> All attributes of the parent class are Lists of objects (some of them >>>> are still not serializable and some of them are, i.e. List<String>). >>>> >>>> Until I do map and filter on the RDD that objects are filled correclty >>>> (I checked that via Eclipse debug), but when I do collect all the >>>> attributes of my objects are empty. Could you help me please? >>>> I'm using spark-core-2.10 e version 0.9.0-incubating. >>>> >>>> Best, >>>> Flavio >>>> >>>> >>> >
