Re: How Beam coders match with runner serialization

2020-05-19 Thread Ivan San Jose
Ok, sorry but now I'm totally lost... Then, as you said "Beam coders are wrapped in Flink's TypeSerializers", so Beam is not doing anything about serialization/deserialization, is leveraging this to Flink TypeSerializers, right? Now let me explain my problem better... We have a Beam pipeline usi

Re: How Beam coders match with runner serialization

2020-05-19 Thread Maximilian Michels
Hi Ivan, Beam does not use Java serialization for checkpoint data. It uses Beam coders which are wrapped in Flink's TypeSerializers. That said, Beam does not support serializer migration yet. I'm curious, what do you consider a "backwards-compatible" change? If you are attempting to upgrade the s

How Beam coders match with runner serialization

2020-05-19 Thread Ivan San Jose
Hi, I've been started to use Apache Beam not so long ago (so bear with me please) and I have a question about Beam coders and how are they related with under the hood runner serializer... Let me explain myself better: As far I've understood from Beam documentation, coders are being used in order t