[
https://issues.apache.org/jira/browse/FLINK-30863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17699585#comment-17699585
]
jinghaihang commented on FLINK-30863:
-------------------------------------
Hello Yanfei Lei, I found another situation that also triggers
FileNotFoundException in local recvoery mode, which has nothing to do with the
materialization process.
The specific description is as follows:
When a checkpoint is completed, the sendAcknowledgeMessages method will be
called to notify each ExecutionVertex that the checkpoint is completed, and
finally FsStateChangelogWriter#confirm() is called. When there are two or more
operators in one tm, one operator will accidentally delete the changelog local
files of other operators when executing the confirm() method.
[~Yanfei Lei] Please help to confirm whether this issue is discussed in this
issue, or I am raising a separate issue？Thanks
> Register local recovery files of changelog before notifyCheckpointComplete()
> ----------------------------------------------------------------------------
>
> Key: FLINK-30863
> URL: https://issues.apache.org/jira/browse/FLINK-30863
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / State Backends
> Affects Versions: 1.17.0
> Reporter: Yanfei Lei
> Assignee: Yanfei Lei
> Priority: Major
> Labels: pull-request-available
> Attachments: tm-log_fail_cl_local_recovery.txt
>
>
> If TM is materialized before receiving confirm(), the previously uploaded
> queue in `FsStateChangelogWriter` will be cleared, so the local files of the
> completed checkpoint will not be registered again, while the JM owned files
> are registered before confirm(), and do not depend on the uploaded queue, so
> the local files are deleted, and the DFS files are still there.
>
> We have encountered the following situation, the job cannot find the local
> recovery files, but can restore from the DFS files:
> {code:java}
> 2023-01-18 17:21:13,412 [SlidingProcessingTimeWindows (37/48)#1] INFO
> org.apache.flink.runtime.taskmanager.Task [] -
> SlidingProcessingTimeWindows (37/48)#1 #1 (fa12cfa3b811a351e031b036b0e85d91)
> switched from DEPLOYING to INITIALIZING.
> 2023-01-18 17:21:13,440 [SlidingProcessingTimeWindows (37/48)#1] INFO
> org.apache.flink.runtime.state.TaskLocalStateStoreImpl [] - Found
> registered local state for checkpoint 11599 in subtask
> (2daf1d9bc9ed40ecb191303db813b0de - 0a448493b4782967b150582570326227 - 36) :
> TaskOperatorSubtaskStates{subtaskStatesByOperatorID={0a448493b4782967b150582570326227=SubtaskState{operatorStateFromBackend=StateObjectCollection{[]},
> operatorStateFromStream=StateObjectCollection{[]},
> keyedStateFromBackend=StateObjectCollection{[org.apache.flink.runtime.state.changelog.ChangelogStateBackendLocalHandle@38aa46db]},
> keyedStateFromStream=StateObjectCollection{[]},
> inputChannelState=StateObjectCollection{[]},
> resultSubpartitionState=StateObjectCollection{[]}, stateSize=1764644202,
> checkpointedSize=1997682}}, isTaskDeployedAsFinished=false,
> isTaskFinished=false}
> 2023-01-18 17:21:13,442 [SlidingProcessingTimeWindows (37/48)#1] INFO
> org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend [] -
> Getting managed memory shared cache for RocksDB.
> 2023-01-18 17:21:13,446 [SlidingProcessingTimeWindows (37/48)#1] INFO
> org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend [] -
> Obtained shared RocksDB cache of size 1438814063 bytes
> 2023-01-18 17:21:13,447 [SlidingProcessingTimeWindows (37/48)#1] INFO
> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation
> [] - Starting to restore from state handle:
> IncrementalLocalKeyedStateHandle{metaDataState=File State:
> file:/opt/flink/flink-tmp-dir/tm_job-2daf1d9b-c9ed-40ec-b191-303db813b0de-taskmanager-1-31/localState/aid_45af7e6b612dad10b60554d81323d5f3/jid_2daf1d9bc9ed40ecb191303db813b0de/vtx_0a448493b4782967b150582570326227_sti_36/chk_125/0d082666-bd31-4ebe-9977-545c0d9b18a5
> [1187 bytes]}
> DirectoryKeyedStateHandle{directoryStateHandle=DirectoryStateHandle{directory=/opt/flink/flink-tmp-dir/tm_job-2daf1d9b-c9ed-40ec-b191-303db813b0de-taskmanager-1-31/localState/aid_45af7e6b612dad10b60554d81323d5f3/jid_2daf1d9bc9ed40ecb191303db813b0de/vtx_0a448493b4782967b150582570326227_sti_36/chk_125/b3e1d20f164d4c5baed291f5d1224183},
> keyGroupRange=KeyGroupRange{startKeyGroup=96, endKeyGroup=98}} without
> rescaling.
> 2023-01-18 17:21:13,495 [SlidingProcessingTimeWindows (37/48)#1] INFO
> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation
> [] - Finished restoring from state handle:
> IncrementalLocalKeyedStateHandle{metaDataState=File State:
> file:/opt/flink/flink-tmp-dir/tm_job-2daf1d9b-c9ed-40ec-b191-303db813b0de-taskmanager-1-31/localState/aid_45af7e6b612dad10b60554d81323d5f3/jid_2daf1d9bc9ed40ecb191303db813b0de/vtx_0a448493b4782967b150582570326227_sti_36/chk_125/0d082666-bd31-4ebe-9977-545c0d9b18a5
> [1187 bytes]}
> DirectoryKeyedStateHandle{directoryStateHandle=DirectoryStateHandle{directory=/opt/flink/flink-tmp-dir/tm_job-2daf1d9b-c9ed-40ec-b191-303db813b0de-taskmanager-1-31/localState/aid_45af7e6b612dad10b60554d81323d5f3/jid_2daf1d9bc9ed40ecb191303db813b0de/vtx_0a448493b4782967b150582570326227_sti_36/chk_125/b3e1d20f164d4c5baed291f5d1224183},
> keyGroupRange=KeyGroupRange{startKeyGroup=96, endKeyGroup=98}} without
> rescaling.
> 2023-01-18 17:21:13,495 [SlidingProcessingTimeWindows (37/48)#1] INFO
> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation
> [] - restore rocksdb cost 48 ms.
> 2023-01-18 17:21:13,495 [SlidingProcessingTimeWindows (37/48)#1] INFO
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder [] -
> Finished building RocksDB keyed state-backend at
> /opt/flink/flink-tmp-dir/tm_job-2daf1d9b-c9ed-40ec-b191-303db813b0de-taskmanager-1-31/tmp/job_2daf1d9bc9ed40ecb191303db813b0de_op_WindowOperator_0a448493b4782967b150582570326227__37_48__uuid_2cbcf5ff-4451-4788-8762-158077c8368e.
> 2023-01-18 17:21:13,501 [SlidingProcessingTimeWindows (37/48)#1] INFO
> org.apache.flink.changelog.fs.FsStateChangelogStorage [] -
> createWriter for operator
> WindowOperator_0a448493b4782967b150582570326227_(37/48)/KeyGroupRange{startKeyGroup=96,
> endKeyGroup=98}: 00000000-0000-0000-0000-000000000001
> 2023-01-18 17:21:13,502 [SlidingProcessingTimeWindows (37/48)#1] INFO
> org.apache.flink.state.changelog.restore.ChangelogBackendRestoreOperation []
> - read changelog handle start, total state size=190851072 .
> 2023-01-18 17:21:13,502 [SlidingProcessingTimeWindows (37/48)#1] INFO
> org.apache.flink.runtime.state.changelog.StateChangelogStorageLoader [] -
> Creating a changelog storage with name 'filesystem' to restore from
> 'ChangelogStateHandleStreamImpl'.
> 2023-01-18 17:21:13,529 [Source Data Fetcher for Source: KafkaWindowSource
> (37/48)#1] INFO org.apache.kafka.clients.Metadata
> [] - [Consumer clientId=xr_cl_1-36, groupId=xr_cl_1] Cluster ID:
> 56sVc6RESJ63Jh6BnsMjkA
> 2023-01-18 17:21:13,515 [SlidingProcessingTimeWindows (37/48)#1] WARN
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure [] -
> Exception while restoring keyed state backend for
> WindowOperator_0a448493b4782967b150582570326227_(37/48) from alternative
> (1/2), will retry while more alternatives are available.
> java.lang.RuntimeException: java.io.FileNotFoundException:
> /opt/flink/flink-tmp-dir/tm_job-2daf1d9b-c9ed-40ec-b191-303db813b0de-taskmanager-1-31/localState/aid_45af7e6b612dad10b60554d81323d5f3/jid_2daf1d9bc9ed40ecb191303db813b0de/taskowned/cc3bac5d-020c-4ee0-8999-d661f4b9beac
> (No such file or directory)
> at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321)
> ~[flink-dist_2.12-1.15-vvr-6.0-SNAPSHOT.jar:1.15-vvr-6.0-SNAPSHOT]
> at
> org.apache.flink.runtime.state.changelog.StateChangelogHandleStreamHandleReader$1.advance(StateChangelogHandleStreamHandleReader.java:87)
> ~[flink-dist_2.12-1.15-vvr-6.0-SNAPSHOT.jar:1.15-vvr-6.0-SNAPSHOT]
> at
> org.apache.flink.runtime.state.changelog.StateChangelogHandleStreamHandleReader$1.hasNext(StateChangelogHandleStreamHandleReader.java:69)
> ~[flink-dist_2.12-1.15-vvr-6.0-SNAPSHOT.jar:1.15-vvr-6.0-SNAPSHOT]
> at
> org.apache.flink.state.changelog.restore.ChangelogBackendRestoreOperation.readBackendHandle(ChangelogBackendRestoreOperation.java:121)
> ~[flink-dist_2.12-1.15-vvr-6.0-SNAPSHOT.jar:1.15-vvr-6.0-SNAPSHOT]
> at
> org.apache.flink.state.changelog.restore.ChangelogBackendRestoreOperation.restore(ChangelogBackendRestoreOperation.java:89)
> ~[flink-dist_2.12-1.15-vvr-6.0-SNAPSHOT.jar:1.15-vvr-6.0-SNAPSHOT]
> at
> org.apache.flink.state.changelog.ChangelogStateBackend.restore(ChangelogStateBackend.java:94)
> ~[flink-dist_2.12-1.15-vvr-6.0-SNAPSHOT.jar:1.15-vvr-6.0-SNAPSHOT]
> at
> org.apache.flink.state.changelog.AbstractChangelogStateBackend.createKeyedStateBackend(AbstractChangelogStateBackend.java:136)
> ~[flink-dist_2.12-1.15-vvr-6.0-SNAPSHOT.jar:1.15-vvr-6.0-SNAPSHOT]
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:336)
> ~[flink-dist_2.12-1.15-vvr-6.0-SNAPSHOT.jar:1.15-vvr-6.0-SNAPSHOT]
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:168)
> ~[flink-dist_2.12-1.15-vvr-6.0-SNAPSHOT.jar:1.15-vvr-6.0-SNAPSHOT]
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)
> ~[flink-dist_2.12-1.15-vvr-6.0-SNAPSHOT.jar:1.15-vvr-6.0-SNAPSHOT]
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:353)
> ~[flink-dist_2.12-1.15-vvr-6.0-SNAPSHOT.jar:1.15-vvr-6.0-SNAPSHOT]
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:165)
> ~[flink-dist_2.12-1.15-vvr-6.0-SNAPSHOT.jar:1.15-vvr-6.0-SNAPSHOT]
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:267)
> ~[flink-dist_2.12-1.15-vvr-6.0-SNAPSHOT.jar:1.15-vvr-6.0-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:106)
> ~[flink-dist_2.12-1.15-vvr-6.0-SNAPSHOT.jar:1.15-vvr-6.0-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:701)
> ~[flink-dist_2.12-1.15-vvr-6.0-SNAPSHOT.jar:1.15-vvr-6.0-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
> ~[flink-dist_2.12-1.15-vvr-6.0-SNAPSHOT.jar:1.15-vvr-6.0-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:677)
> ~[flink-dist_2.12-1.15-vvr-6.0-SNAPSHOT.jar:1.15-vvr-6.0-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:644)
> ~[flink-dist_2.12-1.15-vvr-6.0-SNAPSHOT.jar:1.15-vvr-6.0-SNAPSHOT]
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:954)
> [flink-dist_2.12-1.15-vvr-6.0-SNAPSHOT.jar:1.15-vvr-6.0-SNAPSHOT]
> at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:923)
> [flink-dist_2.12-1.15-vvr-6.0-SNAPSHOT.jar:1.15-vvr-6.0-SNAPSHOT]
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:746)
> [flink-dist_2.12-1.15-vvr-6.0-SNAPSHOT.jar:1.15-vvr-6.0-SNAPSHOT]
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:568)
> [flink-dist_2.12-1.15-vvr-6.0-SNAPSHOT.jar:1.15-vvr-6.0-SNAPSHOT]
> at java.lang.Thread.run(Thread.java:834) [?:1.8.0_102]
> Caused by: java.io.FileNotFoundException:
> /opt/flink/flink-tmp-dir/tm_job-2daf1d9b-c9ed-40ec-b191-303db813b0de-taskmanager-1-31/localState/aid_45af7e6b612dad10b60554d81323d5f3/jid_2daf1d9bc9ed40ecb191303db813b0de/taskowned/cc3bac5d-020c-4ee0-8999-d661f4b9beac
> (No such file or directory)
> at java.io.FileInputStream.open0(Native Method) ~[?:1.8.0_102]
> at java.io.FileInputStream.open(FileInputStream.java:195) ~[?:1.8.0_102]
> at java.io.FileInputStream.<init>(FileInputStream.java:138) ~[?:1.8.0_102]
> at
> org.apache.flink.core.fs.local.LocalDataInputStream.<init>(LocalDataInputStream.java:50)
> ~[flink-dist_2.12-1.15-vvr-6.0-SNAPSHOT.jar:1.15-vvr-6.0-SNAPSHOT]
> at
> org.apache.flink.core.fs.local.LocalFileSystem.open(LocalFileSystem.java:141)
> ~[flink-dist_2.12-1.15-vvr-6.0-SNAPSHOT.jar:1.15-vvr-6.0-SNAPSHOT]
> at
> org.apache.flink.core.fs.SafetyNetWrapperFileSystem.open(SafetyNetWrapperFileSystem.java:89)
> ~[flink-dist_2.12-1.15-vvr-6.0-SNAPSHOT.jar:1.15-vvr-6.0-SNAPSHOT]
> at
> org.apache.flink.runtime.state.filesystem.FileStateHandle.openInputStream(FileStateHandle.java:72)
> ~[flink-dist_2.12-1.15-vvr-6.0-SNAPSHOT.jar:1.15-vvr-6.0-SNAPSHOT]
> at
> org.apache.flink.changelog.fs.ChangelogStreamHandleReaderWithCache.openAndSeek(ChangelogStreamHandleReaderWithCache.java:89)
> ~[flink-dist_2.12-1.15-vvr-6.0-SNAPSHOT.jar:1.15-vvr-6.0-SNAPSHOT]
> at
> org.apache.flink.changelog.fs.StateChangeIteratorImpl.read(StateChangeIteratorImpl.java:42)
> ~[flink-dist_2.12-1.15-vvr-6.0-SNAPSHOT.jar:1.15-vvr-6.0-SNAPSHOT]
> at
> org.apache.flink.runtime.state.changelog.StateChangelogHandleStreamHandleReader$1.advance(StateChangelogHandleStreamHandleReader.java:85)
> ~[flink-dist_2.12-1.15-vvr-6.0-SNAPSHOT.jar:1.15-vvr-6.0-SNAPSHOT]
> ... 21 more
> 2023-01-18 17:21:13,545 [SlidingProcessingTimeWindows (37/48)#1] INFO
> org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend [] -
> Getting managed memory shared cache for RocksDB.
> 2023-01-18 17:21:13,545 [SlidingProcessingTimeWindows (37/48)#1] INFO
> org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend [] -
> Obtained shared RocksDB cache of size 1438814063 bytes
> 2023-01-18 17:21:13,546 [SlidingProcessingTimeWindows (37/48)#1] INFO
> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation
> [] - Starting to restore from state handle:
> IncrementalRemoteKeyedStateHandle{backendIdentifier=b3e1d20f-164d-4c5b-aed2-91f5d1224183,
> stateHandleId=f404ffdb-715e-4f95-a850-f459639a30e6,
> keyGroupRange=KeyGroupRange{startKeyGroup=96, endKeyGroup=98},
> checkpointId=125, sharedState={001388.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/bd7103e0-fe66-4400-a2a2-e4f3dda01b71
> [51353250 bytes], 001383.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/5e937d12-e1d5-4427-9401-80f5db6af2ee
> [67393725 bytes], 001314.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/c2d374dc-88ba-4003-bad5-81590e56963d
> [67407704 bytes], 001403.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/a7ec0f13-3ef1-4e69-a50e-071b9f6b092b
> [67411564 bytes], 001416.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/9c82f81a-33d6-4b8f-a237-2150d0c311d8
> [10391374 bytes], 001384.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/b591513d-b850-4025-a126-452137b4a6fa
> [67397014 bytes], 001413.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/5e2a0408-f9b4-4875-8594-2dc59df3bc66
> [5307477 bytes], 001400.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/16bf5cda-4f04-40c9-8b54-ca1c393028f9
> [67953551 bytes], 001316.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/9543bc5c-dd9d-47b6-9626-159559b9ee45
> [67406146 bytes], 001315.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/b6099b46-bbef-486a-8c2f-ef232eefdeff
> [67409984 bytes], 001408.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/66354b67-97b2-41e8-ae48-678af553d2a7
> [16861835 bytes], 001404.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/a550c63b-003c-48d5-9bc2-45341ce4e641
> [67413763 bytes], 001406.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/55822a0c-8448-48d2-a0c1-0bbccfaf31ca
> [67414722 bytes], 001317.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/2b1f4dcc-76e2-4492-8470-bd512031a0ab
> [67407862 bytes], 001401.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/42a7b773-95d6-4a8b-bb18-64f083e9c627
> [20184247 bytes], 001414.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/ec3a3d64-c0c1-4a5e-b541-dfc01a0f028d
> [19636706 bytes], 001385.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/2ef14224-f229-4852-b97e-50268979f2a3
> [67395048 bytes], 001399.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/a4550160-1c0c-4bb7-9eb9-26e2a6d9ed43
> [67931628 bytes], 001381.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/c6278757-9048-4e9d-a67d-5b9fd46c2c4f
> [67404982 bytes],
> 001368.sst=ByteStreamStateHandle{handleName='oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/4497322c-c46b-4d42-9313-1f36d929c577',
> dataBytes=1354}, 001407.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/3308882e-bf0f-4f26-9f3a-3d3dfb05fc85
> [67409775 bytes], 001405.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/5d5e1529-b109-483d-afa6-5d44297bec6d
> [67415504 bytes], 001318.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/7f9a7aa5-db80-4448-b987-1f9fdb88f5d1
> [67406909 bytes], 001386.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/2cf3a080-37d5-4133-b9d2-83e2ad49fc24
> [67395104 bytes], 001411.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/66d3176b-2257-403a-be82-03c49162c05f
> [19605638 bytes], 001410.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/108bb1c1-99da-4920-acab-15f254326ed0
> [11733963 bytes], 001387.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/9027f480-b895-4336-80b9-dea244ce1572
> [67397047 bytes], 001382.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/398f2188-6336-4efe-a014-6bd3bf2ce8c6
> [67396271 bytes], 001415.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/c73af6f6-18e2-424e-90f9-586809dfbba5
> [1931553 bytes], 001402.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/f80462bb-eef1-4e3a-845c-97dea160e306
> [67410281 bytes], 001313.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/88251386-e092-4a5e-b8bf-9c2e65148f5a
> [67408879 bytes]},
> privateState={OPTIONS-000013=ByteStreamStateHandle{handleName='oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/b32ad6b5-4540-4b3e-9a82-14cebe231898',
> dataBytes=17286}, MANIFEST-000004=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/c94b6317-b50b-4bf1-938d-83d7aad6ed6a
> [179781 bytes],
> CURRENT=ByteStreamStateHandle{handleName='oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/3665ebc2-52ee-4a2e-87b6-6e0cc8458c37',
> dataBytes=16}},
> metaStateHandle=ByteStreamStateHandle{handleName='oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/9af5edc4-c589-4f0b-91d7-515e068a3454',
> dataBytes=1187}, registered=false} without rescaling.
> 2023-01-18 17:22:08,867 [SlidingProcessingTimeWindows (37/48)#1] INFO
> org.apache.flink.contrib.streaming.state.RocksDBStateDownloader [] - download
> IncrementalRemoteKeyedStateHandle{backendIdentifier=b3e1d20f-164d-4c5b-aed2-91f5d1224183,
> stateHandleId=f404ffdb-715e-4f95-a850-f459639a30e6,
> keyGroupRange=KeyGroupRange{startKeyGroup=96, endKeyGroup=98},
> checkpointId=125, sharedState={001388.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/bd7103e0-fe66-4400-a2a2-e4f3dda01b71
> [51353250 bytes], 001383.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/5e937d12-e1d5-4427-9401-80f5db6af2ee
> [67393725 bytes], 001314.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/c2d374dc-88ba-4003-bad5-81590e56963d
> [67407704 bytes], 001403.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/a7ec0f13-3ef1-4e69-a50e-071b9f6b092b
> [67411564 bytes], 001416.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/9c82f81a-33d6-4b8f-a237-2150d0c311d8
> [10391374 bytes], 001384.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/b591513d-b850-4025-a126-452137b4a6fa
> [67397014 bytes], 001413.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/5e2a0408-f9b4-4875-8594-2dc59df3bc66
> [5307477 bytes], 001400.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/16bf5cda-4f04-40c9-8b54-ca1c393028f9
> [67953551 bytes], 001316.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/9543bc5c-dd9d-47b6-9626-159559b9ee45
> [67406146 bytes], 001315.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/b6099b46-bbef-486a-8c2f-ef232eefdeff
> [67409984 bytes], 001408.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/66354b67-97b2-41e8-ae48-678af553d2a7
> [16861835 bytes], 001404.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/a550c63b-003c-48d5-9bc2-45341ce4e641
> [67413763 bytes], 001406.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/55822a0c-8448-48d2-a0c1-0bbccfaf31ca
> [67414722 bytes], 001317.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/2b1f4dcc-76e2-4492-8470-bd512031a0ab
> [67407862 bytes], 001401.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/42a7b773-95d6-4a8b-bb18-64f083e9c627
> [20184247 bytes], 001414.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/ec3a3d64-c0c1-4a5e-b541-dfc01a0f028d
> [19636706 bytes], 001385.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/2ef14224-f229-4852-b97e-50268979f2a3
> [67395048 bytes], 001399.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/a4550160-1c0c-4bb7-9eb9-26e2a6d9ed43
> [67931628 bytes], 001381.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/c6278757-9048-4e9d-a67d-5b9fd46c2c4f
> [67404982 bytes],
> 001368.sst=ByteStreamStateHandle{handleName='oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/4497322c-c46b-4d42-9313-1f36d929c577',
> dataBytes=1354}, 001407.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/3308882e-bf0f-4f26-9f3a-3d3dfb05fc85
> [67409775 bytes], 001405.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/5d5e1529-b109-483d-afa6-5d44297bec6d
> [67415504 bytes], 001318.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/7f9a7aa5-db80-4448-b987-1f9fdb88f5d1
> [67406909 bytes], 001386.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/2cf3a080-37d5-4133-b9d2-83e2ad49fc24
> [67395104 bytes], 001411.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/66d3176b-2257-403a-be82-03c49162c05f
> [19605638 bytes], 001410.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/108bb1c1-99da-4920-acab-15f254326ed0
> [11733963 bytes], 001387.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/9027f480-b895-4336-80b9-dea244ce1572
> [67397047 bytes], 001382.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/398f2188-6336-4efe-a014-6bd3bf2ce8c6
> [67396271 bytes], 001415.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/c73af6f6-18e2-424e-90f9-586809dfbba5
> [1931553 bytes], 001402.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/f80462bb-eef1-4e3a-845c-97dea160e306
> [67410281 bytes], 001313.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/88251386-e092-4a5e-b8bf-9c2e65148f5a
> [67408879 bytes]},
> privateState={OPTIONS-000013=ByteStreamStateHandle{handleName='oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/b32ad6b5-4540-4b3e-9a82-14cebe231898',
> dataBytes=17286}, MANIFEST-000004=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/c94b6317-b50b-4bf1-938d-83d7aad6ed6a
> [179781 bytes],
> CURRENT=ByteStreamStateHandle{handleName='oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/3665ebc2-52ee-4a2e-87b6-6e0cc8458c37',
> dataBytes=16}},
> metaStateHandle=ByteStreamStateHandle{handleName='oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/9af5edc4-c589-4f0b-91d7-515e068a3454',
> dataBytes=1187}, registered=false}, state size = 1573793130, cost 55319 ms.
> 2023-01-18 17:22:08,909 [SlidingProcessingTimeWindows (37/48)#1] INFO
> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation
> [] - Finished restoring from state handle:
> IncrementalRemoteKeyedStateHandle{backendIdentifier=b3e1d20f-164d-4c5b-aed2-91f5d1224183,
> stateHandleId=f404ffdb-715e-4f95-a850-f459639a30e6,
> keyGroupRange=KeyGroupRange{startKeyGroup=96, endKeyGroup=98},
> checkpointId=125, sharedState={001388.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/bd7103e0-fe66-4400-a2a2-e4f3dda01b71
> [51353250 bytes], 001383.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/5e937d12-e1d5-4427-9401-80f5db6af2ee
> [67393725 bytes], 001314.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/c2d374dc-88ba-4003-bad5-81590e56963d
> [67407704 bytes], 001403.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/a7ec0f13-3ef1-4e69-a50e-071b9f6b092b
> [67411564 bytes], 001416.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/9c82f81a-33d6-4b8f-a237-2150d0c311d8
> [10391374 bytes], 001384.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/b591513d-b850-4025-a126-452137b4a6fa
> [67397014 bytes], 001413.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/5e2a0408-f9b4-4875-8594-2dc59df3bc66
> [5307477 bytes], 001400.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/16bf5cda-4f04-40c9-8b54-ca1c393028f9
> [67953551 bytes], 001316.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/9543bc5c-dd9d-47b6-9626-159559b9ee45
> [67406146 bytes], 001315.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/b6099b46-bbef-486a-8c2f-ef232eefdeff
> [67409984 bytes], 001408.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/66354b67-97b2-41e8-ae48-678af553d2a7
> [16861835 bytes], 001404.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/a550c63b-003c-48d5-9bc2-45341ce4e641
> [67413763 bytes], 001406.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/55822a0c-8448-48d2-a0c1-0bbccfaf31ca
> [67414722 bytes], 001317.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/2b1f4dcc-76e2-4492-8470-bd512031a0ab
> [67407862 bytes], 001401.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/42a7b773-95d6-4a8b-bb18-64f083e9c627
> [20184247 bytes], 001414.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/ec3a3d64-c0c1-4a5e-b541-dfc01a0f028d
> [19636706 bytes], 001385.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/2ef14224-f229-4852-b97e-50268979f2a3
> [67395048 bytes], 001399.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/a4550160-1c0c-4bb7-9eb9-26e2a6d9ed43
> [67931628 bytes], 001381.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/c6278757-9048-4e9d-a67d-5b9fd46c2c4f
> [67404982 bytes],
> 001368.sst=ByteStreamStateHandle{handleName='oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/4497322c-c46b-4d42-9313-1f36d929c577',
> dataBytes=1354}, 001407.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/3308882e-bf0f-4f26-9f3a-3d3dfb05fc85
> [67409775 bytes], 001405.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/5d5e1529-b109-483d-afa6-5d44297bec6d
> [67415504 bytes], 001318.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/7f9a7aa5-db80-4448-b987-1f9fdb88f5d1
> [67406909 bytes], 001386.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/2cf3a080-37d5-4133-b9d2-83e2ad49fc24
> [67395104 bytes], 001411.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/66d3176b-2257-403a-be82-03c49162c05f
> [19605638 bytes], 001410.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/108bb1c1-99da-4920-acab-15f254326ed0
> [11733963 bytes], 001387.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/9027f480-b895-4336-80b9-dea244ce1572
> [67397047 bytes], 001382.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/398f2188-6336-4efe-a014-6bd3bf2ce8c6
> [67396271 bytes], 001415.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/c73af6f6-18e2-424e-90f9-586809dfbba5
> [1931553 bytes], 001402.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/f80462bb-eef1-4e3a-845c-97dea160e306
> [67410281 bytes], 001313.sst=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/88251386-e092-4a5e-b8bf-9c2e65148f5a
> [67408879 bytes]},
> privateState={OPTIONS-000013=ByteStreamStateHandle{handleName='oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/b32ad6b5-4540-4b3e-9a82-14cebe231898',
> dataBytes=17286}, MANIFEST-000004=File State:
> oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/c94b6317-b50b-4bf1-938d-83d7aad6ed6a
> [179781 bytes],
> CURRENT=ByteStreamStateHandle{handleName='oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/3665ebc2-52ee-4a2e-87b6-6e0cc8458c37',
> dataBytes=16}},
> metaStateHandle=ByteStreamStateHandle{handleName='oss://cluster-serving/flink-jobs/namespaces/state-test-default/deployments/1b1f8910-047f-4e51-a1bc-eea91e57600d/checkpoints/jobs/2daf1d9b-c9ed-40ec-b191-303db813b0de/2daf1d9bc9ed40ecb191303db813b0de/taskowned/9af5edc4-c589-4f0b-91d7-515e068a3454',
> dataBytes=1187}, registered=false} without rescaling.
> 2023-01-18 17:22:08,911 [SlidingProcessingTimeWindows (37/48)#1] INFO
> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation
> [] - restore rocksdb cost 55365 ms.
> 2023-01-18 17:22:08,912 [SlidingProcessingTimeWindows (37/48)#1] INFO
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder [] -
> Finished building RocksDB keyed state-backend at
> /opt/flink/flink-tmp-dir/tm_job-2daf1d9b-c9ed-40ec-b191-303db813b0de-taskmanager-1-31/tmp/job_2daf1d9bc9ed40ecb191303db813b0de_op_WindowOperator_0a448493b4782967b150582570326227__37_48__uuid_1404e597-c96c-4d7c-99b7-303fd98f80bd.
> 2023-01-18 17:22:08,915 [SlidingProcessingTimeWindows (37/48)#1] WARN
> org.apache.flink.metrics.MetricGroup [] - Name
> collision: Group already contains a Metric with the name
> 'ChangelogStateBackend.lastFullSizeOfMaterialization'. Metric will not be
> reported.[192.168.32.162, taskmanager,
> job-2daf1d9b-c9ed-40ec-b191-303db813b0de-taskmanager-1-31, Flink Streaming
> Job, SlidingProcessingTimeWindows, 36]
> 2023-01-18 17:22:08,915 [SlidingProcessingTimeWindows (37/48)#1] WARN
> org.apache.flink.metrics.MetricGroup [] - Name
> collision: Group already contains a Metric with the name
> 'ChangelogStateBackend.lastIncSizeOfMaterialization'. Metric will not be
> reported.[192.168.32.162, taskmanager,
> job-2daf1d9b-c9ed-40ec-b191-303db813b0de-taskmanager-1-31, Flink Streaming
> Job, SlidingProcessingTimeWindows, 36]
> 2023-01-18 17:22:08,915 [SlidingProcessingTimeWindows (37/48)#1] WARN
> org.apache.flink.metrics.MetricGroup [] - Name
> collision: Group already contains a Metric with the name
> 'ChangelogStateBackend.lastFullSizeOfNonMaterialization'. Metric will not be
> reported.[192.168.32.162, taskmanager,
> job-2daf1d9b-c9ed-40ec-b191-303db813b0de-taskmanager-1-31, Flink Streaming
> Job, SlidingProcessingTimeWindows, 36]
> 2023-01-18 17:22:08,915 [SlidingProcessingTimeWindows (37/48)#1] WARN
> org.apache.flink.metrics.MetricGroup [] - Name
> collision: Group already contains a Metric with the name
> 'ChangelogStateBackend.lastIncSizeOfNonMaterialization'. Metric will not be
> reported.[192.168.32.162, taskmanager,
> job-2daf1d9b-c9ed-40ec-b191-303db813b0de-taskmanager-1-31, Flink Streaming
> Job, SlidingProcessingTimeWindows, 36]
> 2023-01-18 17:22:08,915 [SlidingProcessingTimeWindows (37/48)#1] INFO
> org.apache.flink.changelog.fs.FsStateChangelogStorage [] -
> createWriter for operator
> WindowOperator_0a448493b4782967b150582570326227_(37/48)/KeyGroupRange{startKeyGroup=96,
> endKeyGroup=98}: 00000000-0000-0000-0000-000000000002
> 2023-01-18 17:22:08,915 [SlidingProcessingTimeWindows (37/48)#1] INFO
> org.apache.flink.state.changelog.restore.ChangelogBackendRestoreOperation []
> - read changelog handle start, total state size=190851072 .
> 2023-01-18 17:22:08,919 [SlidingProcessingTimeWindows (37/48)#1] INFO
> org.apache.flink.fs.osshadoop.StsFetcherCredentialsProvider [] - Old
> credential is going to expire. Fetch a new one.
> 2023-01-18 17:22:38,158 [SlidingProcessingTimeWindows (37/48)#1] INFO
> org.apache.flink.state.changelog.restore.ChangelogBackendRestoreOperation []
> - read read changelog handle end, cost 29243 ms.
> 2023-01-18 17:22:38,158 [SlidingProcessingTimeWindows (37/48)#1] INFO
> org.apache.flink.state.common.PeriodicMaterializationManager [] - Task
> SlidingProcessingTimeWindows (37/48)#1 starts periodic materialization
> 2023-01-18 17:22:38,158 [SlidingProcessingTimeWindows (37/48)#1] INFO
> org.apache.flink.state.common.PeriodicMaterializationManager [] - Task
> SlidingProcessingTimeWindows (37/48)#1 schedules the next materialization in
> 82 seconds
> 2023-01-18 17:22:38,176 [SlidingProcessingTimeWindows (37/48)#1] INFO
> org.apache.flink.runtime.taskmanager.Task [] -
> SlidingProcessingTimeWindows (37/48)#1 #1 (fa12cfa3b811a351e031b036b0e85d91)
> switched from INITIALIZING to RUNNING.
> 2023-01-18 17:22:39,057 [SlidingProcessingTimeWindows (37/48)#1] INFO
> org.apache.flink.state.changelog.ChangelogKeyedStateBackend [] - snapshot of
> SlidingProcessingTimeWindows (37/48)#1 for checkpoint 11601, change range:
> 0..2, materialization ID 125
> 2023-01-18 17:22:43,779 [Source Data Fetcher for Source: KafkaWindowSource
> (37/48)#1] INFO
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator [] -
> [Consumer clientId=xr_cl_1-36, groupId=xr_cl_1] Discovered group coordinator
> 192.168.47.158:9092 (id: 2147483546 rack: null)
> 2023-01-18 17:22:44,100 [SlidingProcessingTimeWindows (37/48)#1] INFO
> org.apache.flink.state.changelog.ChangelogKeyedStateBackend [] - snapshot of
> SlidingProcessingTimeWindows (37/48)#1 for checkpoint 11602, change range:
> 0..11, materialization ID 125
> 2023-01-18 17:22:47,531 [SlidingProcessingTimeWindows (37/48)#1] INFO
> org.apache.flink.state.changelog.ChangelogKeyedStateBackend [] - snapshot of
> SlidingProcessingTimeWindows (37/48)#1 for checkpoint 11603, change range:
> 0..17, materialization ID 125
> 2023-01-18 17:22:50,837 [SlidingProcessingTimeWindows (37/48)#1] INFO
> org.apache.flink.state.changelog.ChangelogKeyedStateBackend [] - snapshot of
> SlidingProcessingTimeWindows (37/48)#1 for checkpoint 11604, change range:
> 0..21, materialization ID 125
> 2023-01-18 17:22:53,580 [SlidingProcessingTimeWindows (37/48)#1] INFO
> org.apache.flink.state.changelog.ChangelogKeyedStateBackend [] - snapshot of
> SlidingProcessingTimeWindows (37/48)#1 for checkpoint 11605, change range:
> 0..23, materialization ID 125 {code}
> The above log can be simplified to the following scenario:
> {code:java}
> - cp1 trigger: file1,file1'(local)
> - JM: register [file1] to sharedRegistry
> - cp1 complete: stopTracking [file1], register [file1'] to localRegistry
> - cp2 trigger: file1,file1',file2,file2'
> - JM: register [file1,file2] to sharedRegistry
> - cp2 complete: stopTracking [file1, file2], register [file1',file2'] to
> localRegistry
> - cp1 subsume
> - cp3 trigger: file1,file1',file2,file2',file3,file3'
> - materialization: uploaded.clear()
> - JM: register [file1,file2,file3] to sharedRegistry
> - cp3 complete: stopTracking [file3], register [file3] to localRegistry
> - cp2 subsume: [file1', file2'] are discarded
> - if restore from cp3: local file1',file2' are not found{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
