stevenzwu commented on a change in pull request #2109:
URL: https://github.com/apache/iceberg/pull/2109#discussion_r560641566
##########
File path:
flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java
##########
@@ -106,6 +114,9 @@
// All pending checkpoints states for this function.
private static final ListStateDescriptor<SortedMap<Long, byte[]>>
STATE_DESCRIPTOR = buildStateDescriptor();
private transient ListState<SortedMap<Long, byte[]>> checkpointsState;
+ private static final ListStateDescriptor<Map<Long, Long>>
WATERMARK_DESCRIPTOR = new ListStateDescriptor<>(
+ "iceberg-flink-watermark", new
MapTypeInfo<>(BasicTypeInfo.LONG_TYPE_INFO, BasicTypeInfo.LONG_TYPE_INFO));
+ private transient ListState<Map<Long, Long>> watermarkState;
Review comment:
should we define a `MetaData` to hold all checkpointed metadata fields
so that we don't have to define a new state for each case?
Ideally, I would prefer the metadata and the manifest file bundled in a
single class (per checkpoint). That would require complexity of handling state
schema evolution, which I am not sure if it is worth the effort.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
