Reo-LEI edited a comment on pull request #2862:
URL: https://github.com/apache/iceberg/pull/2862#issuecomment-886186358
Now, the `expireSnapshots` coding is finish, but I'm blocking by a `Kyro
Serialization Error` when I testing.
Kyro raise a NPE when serialize `hadoopConf` of `HadoopFileIO` when flink
create `CombinedScanTask` source. the full stack as follow:
```
Exception in thread "main" java.lang.RuntimeException: Serializing the
source elements failed: java.lang.NullPointerException
Serialization trace:
hadoopConf (org.apache.iceberg.hadoop.HadoopFileIO)
io (org.apache.iceberg.AllManifestsTable$ManifestListReadTask)
tasks (org.apache.iceberg.BaseCombinedScanTask)
at
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.fromCollection(StreamExecutionEnvironment.java:1046)
at
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.fromCollection(StreamExecutionEnvironment.java:1021)
at
org.apache.iceberg.flink.source.MetadataTableSource$Builder.build(MetadataTableSource.java:92)
at
org.apache.iceberg.flink.actions.BaseFlinkAction.loadMetadataTable(BaseFlinkAction.java:117)
at
org.apache.iceberg.flink.actions.BaseFlinkAction.buildValidDataFileTable(BaseFlinkAction.java:92)
at
org.apache.iceberg.flink.actions.ExpireSnapshotsAction.buildValidFileTable(ExpireSnapshotsAction.java:122)
at
org.apache.iceberg.flink.actions.ExpireSnapshotsAction.expire(ExpireSnapshotsAction.java:139)
at
org.apache.iceberg.flink.actions.ExpireSnapshotsAction.execute(ExpireSnapshotsAction.java:117)
at
org.apache.iceberg.flink.actions.ExpireSnapshotsAction.execute(ExpireSnapshotsAction.java:44)
at
cn.walden.stream.IcebergFlinkActions.main(IcebergFlinkActions.java:28)
Caused by: java.io.IOException: Serializing the source elements failed:
java.lang.NullPointerException
Serialization trace:
hadoopConf (org.apache.iceberg.hadoop.HadoopFileIO)
io (org.apache.iceberg.AllManifestsTable$ManifestListReadTask)
tasks (org.apache.iceberg.BaseCombinedScanTask)
at
org.apache.flink.streaming.api.functions.source.FromElementsFunction.<init>(FromElementsFunction.java:93)
at
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.fromCollection(StreamExecutionEnvironment.java:1044)
... 9 more
Caused by: com.esotericsoftware.kryo.KryoException:
java.lang.NullPointerException
Serialization trace:
hadoopConf (org.apache.iceberg.hadoop.HadoopFileIO)
io (org.apache.iceberg.AllManifestsTable$ManifestListReadTask)
tasks (org.apache.iceberg.BaseCombinedScanTask)
at
com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:82)
at
com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
at
com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
at
com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:599)
at
com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:348)
at
com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:289)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
at
com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
at
com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:599)
at
org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.serialize(KryoSerializer.java:316)
at
org.apache.flink.streaming.api.functions.source.FromElementsFunction.<init>(FromElementsFunction.java:89)
... 10 more
Caused by: java.lang.NullPointerException
at
com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:80)
at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:488)
at
com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:57)
... 23 more
```
I'm trying to figure out the reason of that, but I have been blocking for
this for a while and I don't have a clue. Could someone give me a little help?
@rdblue @aokolnychyi @openinx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
