Hi,
Unfortunately, there are currently no compatibility guarantees between
different Beam versions. Beam itself doesn't have the required interfaces or
procedures in place for supporting backwards compatibility of state and there
have been quite some changes in the internals between Flink 1.4 a
Hm, I am sorry to hear this. I must of missed it in docs, that beam version
upgrades can break flink state. It is important information for ones
wanting to use Beam on Flink in production.
So, I guess there is no guarantee for another bump of Flink version to not
break things until it reach 1.7.
E
Flink 1.7 will change the way the "restore serializer" is handled, which
should make it much easier to handle such cases.
Especially breaking java class version format will not be an issue anymore.
That should help to make it easier to give the Beam-on-Flink runner cross
version compatibility.
O
AFAIK the serializer used here is the CoderTypeSerializer which may not
be recoverable because of changes to the contained Coder
(TaggedKvCoder). It doesn't currently have a serialVersionUID, so even
small changes could break serialization backwards-compatibility.
As of now Beam doesn't offer the
Hi Jozef!
When restoring state, the serializer that created the state must still be
available, so the state can be read.
It looks like some serializer classes were removed between Beam versions
(or changed in an incompatible manner).
Backwards compatibility of an operator implementation needs coo
Hello,
I am attempting to upgrade Beam app from 2.5.0 running on Flink 1.4.0 to
Beam 2.6.0 running on Flink 1.5.0. I am not aware of any state migration
changes needed for Flink 1.4.0 -> 1.5.0 so I am just starting a new App
with updated libs from Flink save-point captured by previous version of
