Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/4943
I updated this PR with the following proposed solution:
1. Avro is always part of the user code space, and hence will be loaded
into the user code classloader. This solves multiple problems:
- It allows users to use a different Avro version compared to the
version in a spoiled classpath (for example when Hadoop depenencies pull in
another Avro version).
- It means that Avro class and schema caching is done per user code
classloader, not JVM wide. That prevents "X cannot be cast to X" exceptions.
2. By default, a new pure Avro serializer is used for Avro types (in
flight and in state). Since Avro class format changed, having compatibility
activated by default is not possible anyways.
3. If a user wants to recover an old savepoint that happend to have Avro
types serialized as PoJos, they need to do the following;
- Set the Avro version to the same version that they had when creating
the savepoint. Since Avro now lives in user code space, that is possible
without changing/rebuilding Flink.
- Pass a Flag to the `AvroTypeInfo` to use a backwards compatible
Serializer: `new AvroTypeInfo(myType, true)`.
---
