swapna267 opened a new pull request, #18525: URL: https://github.com/apache/iceberg/pull/18525
PR is to reduce catalog load at job startup from dynamic sink. https://github.com/apache/iceberg/blob/main/flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecordInternalSerializer.java#L103 Here, DynamicRecordInternalSerializer.serialize() has access to Schema and Spec , but did a sanity check on schema id and spec id to eagerly catch any changes to table in between DynamicRecordProcessor and SerializerCache. When DynamicRecordInternal is being deserialized back, schema id lookup anyhow happens to derive the actual schema. If the schema id doesn't exist in underlying table, deserialization fails here instead of previous operator. As the side effect is only pushing the error to downstream operator, this is a better tradeoff to reduce the catalog load on job startup mainly, when the TableSerializer cache is empty. I haven't added any Test in specific here, as existing tests cover the serialization/deserialization of dynamicrecordinternal. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
