curcur commented on a change in pull request #15420:
URL: https://github.com/apache/flink/pull/15420#discussion_r655284210
##########
File path:
flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##########
@@ -114,13 +128,33 @@
private final StateChangelogWriter<?> stateChangelogWriter;
+ private long lastCheckpointId = -1L;
+
/** last accessed partitioned state. */
@SuppressWarnings("rawtypes")
private InternalKvState lastState;
/** For caching the last accessed partitioned state. */
private String lastName;
+ /** Updated initially on restore and later upon materialization (after
FLINK-21356). */
+ private final List<KeyedStateHandle> materialized = new ArrayList<>();
+
+ /** Updated initially on restore and later cleared upon materialization
(after FLINK-21356). */
+ private final List<StateChangelogHandle> nonMaterialized = new
ArrayList<>();
Review comment:
Yes, I think that's a good idea!
And also thanks for clarifying, it totally makes sense.
##########
File path:
flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##########
@@ -245,8 +280,38 @@ public boolean
deregisterKeySelectionListener(KeySelectionListener<K> listener)
@Nonnull CheckpointStreamFactory streamFactory,
@Nonnull CheckpointOptions checkpointOptions)
throws Exception {
- return keyedStateBackend.snapshot(
- checkpointId, timestamp, streamFactory, checkpointOptions);
+ // The range to upload may overlap with the previous one(s). To reuse
them, we could store
+ // the previous results either here in the backend or in the writer.
However,
+ // materialization may truncate only a part of the previous result and
the backend would
+ // have to split it somehow for the former option, so the latter is
used.
Review comment:
addressed offline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
