Hi Thomas, Spark and the Spark runner are using kryo for serialization and it seems to work just fine. What is your exact problem ? stack trace/message ? I've hit an issue with Guava's ImmutableList/Map etc. and used https://github.com/magro/kryo-serializers for that.
For PipelineOptions you can take a look at the Spark runner code here: https://github.com/apache/incubator-beam/blob/master/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkRuntimeContext.java#L73 I'd be happy to assist with Kryo. Thanks, Amit On Wed, Jun 1, 2016 at 7:10 AM Thomas Weise <[email protected]> wrote: > Hi, > > I'm working on putting together a basic runner for Apache Apex. > > Hitting a couple of serialization related issues with running tests. Apex > is using Kryo for serialization by default (and Kryo can delegate to other > serialization frameworks). > > The inner classes of WindowedValue are private and have no default > constructor, which the Kryo field serializer does not like. Also these > classes are not Java serializable, so that's not a fallback option (not > that it would be efficient anyways). > > What's the recommended technique to move the WindowedValues over the wire? > > Also, PipelineOptions aren't serializable, while most other classes are. > They are needed for example with DoFnRunnerBase, so what's the recommended > way to distribute them? Disassemble/reassemble? :) > > Thanks, > Thomas >
