aiborodin commented on code in PR #13394:
URL: https://github.com/apache/iceberg/pull/13394#discussion_r2176752200
##########
flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicIcebergSink.java:
##########
@@ -191,16 +187,11 @@ public static class Builder<T> {
Builder() {}
- public Builder<T> forInput(DataStream<T> inputStream) {
+ public Builder forInput(DataStream<DynamicRecord> inputStream) {
Review Comment:
With no object reuse, the dynamic sink performance test never completes with
changes in this PR and spams with:
```
WARN org.apache.iceberg.flink.sink.dynamic.TableMetadataCache - Performance
degraded as records with different schema is generated for the same table.
Likely the DynamicRecord.schema is not reused. Reuse the same instance if the
record schema is the same to improve performance
WARN org.apache.iceberg.flink.sink.dynamic.TableMetadataCache - Performance
degraded as records with different schema is generated for the same table.
Likely the DynamicRecord.schema is not reused. Reuse the same instance if the
record schema is the same to improve performance
WARN org.apache.iceberg.flink.sink.dynamic.TableMetadataCache - Performance
degraded as records with different schema is generated for the same table.
Likely the DynamicRecord.schema is not reused. Reuse the same instance if the
record schema is the same to improve performance
...
```
I guess this is because schemas are copied across stages. We may indeed need
a custom serialiser to fix this or leave things as status quo. Maybe the
current API is not too bad given it forces object reuse.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
