swapna267 commented on code in PR #18525:
URL: https://github.com/apache/iceberg/pull/18525#discussion_r4383569092
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecordInternalSerializer.java:
##########
@@ -94,21 +94,9 @@ public void serialize(DynamicRecordInternal toSerialize,
DataOutputView dataOutp
}
dataOutputView.writeInt(toSerialize.writerKey());
- final RowDataSerializer rowDataSerializer;
- if (writeSchemaAndSpec) {
- rowDataSerializer =
- serializerCache.serializer(
- toSerialize.tableName(), toSerialize.schema(),
toSerialize.spec());
- } else {
- // Check that the schema id can be resolved. Not strictly necessary for
serialization.
- Tuple3<RowDataSerializer, Schema, PartitionSpec> serializer =
- serializerCache.serializerWithSchemaAndSpec(
- toSerialize.tableName(),
- toSerialize.schema().schemaId(),
- toSerialize.spec().specId());
- rowDataSerializer = serializer.f0;
- }
-
+ RowDataSerializer rowDataSerializer =
+ serializerCache.serializer(
+ toSerialize.tableName(), toSerialize.schema(), toSerialize.spec());
Review Comment:
I think it's better to use separate caches for schema id and schema instead
of shared cache `Map<Object, RowDataSerializer>`
If the schema-keyed side is ever bounded or evicted, it could displace
id-keyed serializers. Separate maps let each have its own lifecycle.
Currently I see [Schema-keyed serializer cache
](https://github.com/apache/iceberg/blob/main/flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/TableSerializerCache.java#L121)
is not bounded. Will change to LRU cache to be safer in worst case scenario of
non-immediate update path and each record carrying a different schema instance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
