kbendick commented on a change in pull request #1185:
URL: https://github.com/apache/iceberg/pull/1185#discussion_r460361491
##########
File path: build.gradle
##########
@@ -253,6 +255,9 @@ project(':iceberg-flink') {
compileOnly("org.apache.hadoop:hadoop-minicluster") {
exclude group: 'org.apache.avro', module: 'avro'
}
+ compileOnly("de.javakaffee:kryo-serializers:0.45") {
+ exclude group: 'com.esotericsoftware', module: 'kryo'
+ }
Review comment:
As discussed in
https://github.com/apache/iceberg/pull/1145/files#diff-22e0fba47887f8c06bc56a654fd3ee9fL363,
we were considering avoiding the usage of `de.javakaffee` for additional kryo
serializers.
Could we create custom Flink serializers, or possibly use POJOs or Flink's
built in support for Avro serializers. There's a recently posted blog post on
the discussion of serializer choices in Flink here:
https://flink.apache.org/news/2020/04/15/flink-serialization-tuning-vol-1.html
I think Avro makes the most sense given that it's been native to Flink for a
long time and because Avro is so widely used in Iceberg already.
As an added bonus, Avro supports full schema evolution of state (such that
changes to the Avro schema of types stored in stateful operators don't cause
said operators to lose their state on upgrade to the new schema, provided they
meet certain specifications which are relatively normal rules for type
migrations, especially with Avro).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
