1996fanrui commented on a change in pull request #15667:
URL: https://github.com/apache/flink/pull/15667#discussion_r616324659
##########
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CompletedCheckpoint.java
##########
@@ -291,6 +306,33 @@ public boolean shouldBeDiscardedOnShutdown(JobStatus
jobStatus) {
jobStatus == JobStatus.SUSPENDED &&
props.discardOnJobSuspended();
}
+ private Set<StreamStateHandle> getSharedStateHandle() {
+ Set<StreamStateHandle> sharedStateHandles = new
HashSet<>(operatorStates.size());
+
+ //reference SavepointV2Serializer.serializeKeyedStateHandle
+ operatorStates.values().forEach(operatorState -> {
+ operatorState.getStates().forEach(operatorSubtaskState
-> {
+ if (operatorSubtaskState != null) {
+ for (KeyedStateHandle managedKeyedState
: operatorSubtaskState.getManagedKeyedState()) {
+ if (managedKeyedState
instanceof IncrementalRemoteKeyedStateHandle) {
+
sharedStateHandles.addAll(((IncrementalRemoteKeyedStateHandle)
managedKeyedState).getSharedState().values());
Review comment:
`((IncrementalRemoteKeyedStateHandle)
managedKeyedState).getSharedState().values()` return all FileStateHandle.
Cleaning up all shared state(FileStateHandle) will cause SharedStateRegistry to
become invalid, and all shared state(FileStateHandle) will be cleaned up?
I used Flink's ITCase to debug and found that
`SharedStateRegistry#unregisterReference` will not be called. Because the
SharedStateRegistry will only take effect when the
`IncrementalRemoteKeyedStateHandle#discardState` method is called.
##########
File path:
flink-core/src/main/java/org/apache/flink/configuration/CheckpointingOptions.java
##########
@@ -147,6 +147,12 @@
.withDescription("The minimum size of state data files.
All state chunks smaller than that are stored" +
" inline in the root checkpoint metadata file.
The max memory threshold for this configuration is 1MB.");
+ /** Flag Whether to enable recursively deletion. If true, can reduce
delete api call in the full checkpoint scenario */
+ public static final ConfigOption<Boolean> ENABLE_DELETE_RECURSIVELY =
ConfigOptions
+ .key("completed.checkpoint.delete.recursive")
+ .defaultValue(true)
+ .withDescription("Whether to enable delete api optimization");
Review comment:
Need to write specific optimization items in the Description
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
