right, if it is not kryo or java serializable then we should allow user to set caching off
2017-08-24 21:55 GMT+03:00 Reuven Lax <re...@google.com.invalid>: > However user classes are not guaranteed to be java serializable either. > > On Thu, Aug 24, 2017 at 11:31 AM, Kobi Salant <kobi.sal...@gmail.com> > wrote: > > > Thank you for your replies, > > > > Lukasz, The kryo does not fail on the WindowedValue but on the user class > > which in some cases is not kryo-serializable usually when there is no > > default constructor. Switching to java serialization does fail on > > WindowedValue > > as it is not defined serializable > > > > Thomas, We thought about writing a custom serializer but it cannot work > > without knowing the exact coders of each RDD, it is similar to the > > discussion that was in the past is it mandatory for the user to set > coders > > for each PCollection or not. This suggestion is like asking Beam to auto > > detect coders for all Pcollections. > > > > Ben, Spark uses java serialization to send closures and it will not work > if > > the cluster contains different JVMs that are not compatible. We agree > with > > you all that it will be more efficient to use coders for caching but that > > means > > a major rewrite of the Spark runner and changing almost all > > methods signatures and internal RDDs from a specific type to byte[]. > > > > As i said before this is fallback for users who have non serializable > user > > classes, even today if an RDD is cached with kryo it doesn't pass the > coder > > so we are not worsening the situation but enabling users to use the Spark > > runner. > > > > Thanks > > Kobi > > > > > > > > בתאריך 24 באוג' 2017 20:37, "Thomas Weise" <t...@apache.org> כתב: > > > > > Would a custom Kryo serializer that uses the coders to perform > > > serialization help? > > > > > > There are various ways Kryo let's you annotate such serializer without > > full > > > surgery, including @Bind on the field level or at class level. > > > > > > Thanks, > > > Thomas > > > > > > > > > On Thu, Aug 24, 2017 at 9:57 AM, Lukasz Cwik <lc...@google.com.invalid > > > > > wrote: > > > > > > > How does Kryo's FieldSerializer fail on WindowedValue/PaneInfo, it > > seems > > > > like those are pretty simple types. > > > > > > > > Also, I don't see why they can't be tagged with Serializable but I > > think > > > > the original reasoning was that coders should be used to guarantee a > > > stable > > > > representation across JVM versions. > > > > > > > > On Thu, Aug 24, 2017 at 5:38 AM, Kobi Salant <kobi.sal...@gmail.com> > > > > wrote: > > > > > > > > > Hi All, > > > > > > > > > > I am working on Spark runner issue > > > > > https://issues.apache.org/jira/browse/BEAM-2669 > > > > > "Kryo serialization exception when DStreams containing > > > > > non-Kryo-serializable data are cached" > > > > > > > > > > Currently, Spark runner enforce the kryo serializer and in > shuffling > > > > > scenarios we always uses coders to transfer bytes over the network. > > But > > > > > Spark can also serialize data for caching/persisting and currently > it > > > > uses > > > > > kryo for this. > > > > > > > > > > Today, when the user uses a class which is not kryo serializable > the > > > > > caching fails and we thought to open the option to use java > > > serialization > > > > > as a fallback. > > > > > > > > > > Our RDDs/DStreams behind the PCollections are usually typed > > > > > RDD/DStream<WindowedValue<InputT>> > > > > > and when Spark tries to java serialize them for caching purposes it > > > fails > > > > > on WindowedValue not being java serializable. > > > > > > > > > > Is there any objection to add Serializable implements to SDK > classes > > > like > > > > > WindowedValue, PaneInfo and others? > > > > > > > > > > Again, i want to emphasise that coders are a big part of the Spark > > > runner > > > > > code and we do use them whenever a shuffle is expected. Changing > the > > > > types > > > > > of the RDDs/DStreams to byte[] will make the code pretty > unreadable > > > and > > > > > will weaken type safety checks. > > > > > > > > > > Thanks > > > > > Kobi > > > > > > > > > > > > > > >
