GitHub user StephanEwen opened a pull request:
https://github.com/apache/flink/pull/4943
[FLINK-6022] [avro] Use Avro to serialize Avro in flight and in State
## What is the purpose of the change
This changes Avro types to be serialized with a proper Avro serializer. The
Avro serializer efficiently handles both Specific Records (generated by Avro)
and Avro-reflection-based serialization.
In order to maintain backwards compatibility, Avro type info generates
actually a wrapping serializer that falls back to a Pojo (or Kryo) serializer
when being reconfigured from an old snapshot.
## Brief change log
- Adds a proper Avro type serializers
- Adds a backwards-compatible Avro serializer that falls back to
Pojo/Kryo on old snapshots
- Adds a bunch of test
## Verifying this change
- Using Avro specific record types in the program and enjoying nice
performant execution ;-)
- Using Avro for Flink state and getting it serialized via Avro, allowing
a schema upgrade of state
- Running the added unit tests
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / **no)**
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (yes / **no)**
- The serializers: **(yes** / no / don't know)
- The runtime per-record code paths (performance sensitive): (yes /
**no** / don't know)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / **no)**
- If yes, how is the feature documented? (**not applicable** / docs /
JavaDocs / not documented)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/StephanEwen/incubator-flink use_proper_avro
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4943.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4943
----
commit dd05a3bf3471702ac8c9129d2d80f2feeca0f949
Author: Stephan Ewen <[email protected]>
Date: 2017-11-03T13:47:33Z
[FLINK-6022] [avro] Use Avro to serialize Avro in flight and in State
This falls back to the original serializer (Pojo / Kryo) in cases where
an old snapshot is resumed.
----
---
