[
https://issues.apache.org/jira/browse/FLINK-31593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17706196#comment-17706196
]
Yanfei Lei commented on FLINK-31593:
------------------------------------
I found the following log in
[^FLINK-31593.StatefulJobSavepointMigrationITCase.create_snapshot.log], files
that cannot be found in restore are deleted during shutdown, If all checkpoints
are considered that it should be discarded on shutdown, then
[lowestRetained|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/StandaloneCompletedCheckpointStore.java#L149-L184]
will be set to {{{}long.max{}}}. I think the
[lowestRetained|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/StandaloneCompletedCheckpointStore.java#L149-L184]
should be set according to `maxNumberOfCheckpointsToRetain`, WDYT? cc
[~roman]
{code:java}
9284 [flink-akka.actor.default-dispatcher-9] DEBUG
org.apache.flink.runtime.state.SharedStateRegistryImpl [] - Discard state
created before checkpoint 9223372036854775807 and not used afterwards
9284 [flink-akka.actor.default-dispatcher-9] DEBUG
org.apache.flink.runtime.state.SharedStateRegistryImpl [] - Scheduled delete of
state handle
Wrapped{IncrementalRemoteKeyedStateHandle{backendIdentifier=813300b3-b788-4d77-a369-cb9d1a8e29c5,
stateHandleId=0063721e-db9a-40d0-9350-2a954e97a6e9,
keyGroupRange=KeyGroupRange{startKeyGroup=3, endKeyGroup=3}, checkpointId=1,
sharedState={}, privateState={MANIFEST-000004=File State:
file:/tmp/junit1099191650015089228/checkpoints_5594496b-29d3-4834-ac07-d7503137513a/b0d7b4f94a20c2f353f65dfd8e8d3682/taskowned/83ba12da-f5d7-4ec6-bc3b-f05aeb5af869
[1504 bytes], 000037.sst=File State:
file:/tmp/junit1099191650015089228/checkpoints_5594496b-29d3-4834-ac07-d7503137513a/b0d7b4f94a20c2f353f65dfd8e8d3682/taskowned/693caeea-6ca5-435b-97b9-6097074b4af3
[978 bytes], 000031.sst=File State:
file:/tmp/junit1099191650015089228/checkpoints_5594496b-29d3-4834-ac07-d7503137513a/b0d7b4f94a20c2f353f65dfd8e8d3682/taskowned/0c7a6f22-2e37-4d4d-b11e-807e7754910b
[985 bytes], 000033.sst=File State:
file:/tmp/junit1099191650015089228/checkpoints_5594496b-29d3-4834-ac07-d7503137513a/b0d7b4f94a20c2f353f65dfd8e8d3682/taskowned/3ff030db-db0b-44f1-b622-412999fd522f
[971 bytes], 000035.sst=File State:
file:/tmp/junit1099191650015089228/checkpoints_5594496b-29d3-4834-ac07-d7503137513a/b0d7b4f94a20c2f353f65dfd8e8d3682/taskowned/398f55fa-819a-4094-adac-aea6064b5a12
[974 bytes], CURRENT=File State:
file:/tmp/junit1099191650015089228/checkpoints_5594496b-29d3-4834-ac07-d7503137513a/b0d7b4f94a20c2f353f65dfd8e8d3682/taskowned/5d13630c-1f30-47c9-8e29-536fce74609a
[16 bytes], 000039.sst=File State:
file:/tmp/junit1099191650015089228/checkpoints_5594496b-29d3-4834-ac07-d7503137513a/b0d7b4f94a20c2f353f65dfd8e8d3682/taskowned/b233c3d8-8654-48a1-bb84-aa3f39d9d98d
[968 bytes], OPTIONS-000029=File State:
file:/tmp/junit1099191650015089228/checkpoints_5594496b-29d3-4834-ac07-d7503137513a/b0d7b4f94a20c2f353f65dfd8e8d3682/taskowned/5429c603-2c50-4ae2-90ae-4be14cb00645
[46014 bytes], 000038.sst=File State:
file:/tmp/junit1099191650015089228/checkpoints_5594496b-29d3-4834-ac07-d7503137513a/b0d7b4f94a20c2f353f65dfd8e8d3682/taskowned/f1c03c83-bd20-4878-9049-23f0e1d94049
[971 bytes], 000030.sst=File State:
file:/tmp/junit1099191650015089228/checkpoints_5594496b-29d3-4834-ac07-d7503137513a/b0d7b4f94a20c2f353f65dfd8e8d3682/taskowned/e64cbdb2-fa10-4d93-bfc1-a25bbd3f78f6
[976 bytes], 000036.sst=File State:
file:/tmp/junit1099191650015089228/checkpoints_5594496b-29d3-4834-ac07-d7503137513a/b0d7b4f94a20c2f353f65dfd8e8d3682/taskowned/7e12fafb-a46a-4af1-bc99-6168fb5f2523
[968 bytes], 000032.sst=File State:
file:/tmp/junit1099191650015089228/checkpoints_5594496b-29d3-4834-ac07-d7503137513a/b0d7b4f94a20c2f353f65dfd8e8d3682/taskowned/f7d435f9-e9fa-496b-8d74-8a2c039e7db4
[975 bytes], 000034.sst=File State:
file:/tmp/junit1099191650015089228/checkpoints_5594496b-29d3-4834-ac07-d7503137513a/b0d7b4f94a20c2f353f65dfd8e8d3682/taskowned/dec0bd1a-b937-4e59-9143-5eeb71de6089
[2099 bytes]}, metaStateHandle=File State:
file:/tmp/junit1099191650015089228/checkpoints_5594496b-29d3-4834-ac07-d7503137513a/b0d7b4f94a20c2f353f65dfd8e8d3682/taskowned/a3f92dc5-1604-4e75-a7ee-f680f5401205
[8135 bytes], registered=true}}. {code}
> Update reference data for Migration Tests
> -----------------------------------------
>
> Key: FLINK-31593
> URL: https://issues.apache.org/jira/browse/FLINK-31593
> Project: Flink
> Issue Type: Sub-task
> Reporter: Matthias Pohl
> Assignee: Matthias Pohl
> Priority: Major
> Labels: pull-request-available
> Attachments:
> FLINK-31593.StatefulJobSavepointMigrationITCase.create_snapshot.log,
> FLINK-31593.StatefulJobSavepointMigrationITCase.verify_snapshot.log
>
>
> # Update {{CURRENT_VERSION in TypeSerializerUpgradeTestBase}} with the new
> version. This will likely fail some tests because snapshots are missing for
> that version. Generate them, for example in
> {{TypeSerializerUpgradeTestBase.}}
> # (major/minor only) Update migration tests in master to cover migration
> from new version: (search for usages of FlinkV{{{}ersion{}}})
> ** AbstractOperatorRestoreTestBase
> ** CEPMigrationTest
> ** BucketingSinkMigrationTest
> ** FlinkKafkaConsumerBaseMigrationTest
> ** ContinuousFileProcessingMigrationTest
> ** WindowOperatorMigrationTest
> ** StatefulJobSavepointMigrationITCase
> ** StatefulJobWBroadcastStateMigrationITCase
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
