[
https://issues.apache.org/jira/browse/FLINK-31593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17706830#comment-17706830
]
Matthias Pohl commented on FLINK-31593:
---------------------------------------
It feels like the previous try was a lucky shot. I wanted to do the same with
{{StatefulJobWBroadcastStateMigrationITCase}} since it's also failing. But
there are constantly some incremental checkpoints that are cleaned up based on
the logs:
{code}
14072 [flink-akka.actor.default-dispatcher-8] DEBUG
org.apache.flink.runtime.state.SharedStateRegistryImpl [] - Discard state
created before checkpoint 9223372036854775807 and not used afterwards
14072 [flink-akka.actor.default-dispatcher-8] DEBUG
org.apache.flink.runtime.state.SharedStateRegistryImpl [] - Scheduled delete of
state handle
Wrapped{IncrementalRemoteKeyedStateHandle{backendIdentifier=a8221137-35b5-4a81-b6b6-dbd42f26b72e,
stateHandleId=f6d026b2-070c-4f65-906b-9c8a76f464a2,
keyGroupRange=KeyGroupRange{startKeyGroup=1, endKeyGroup=1}, checkpointId=1,
sharedState={}, privateState={MANIFEST-000004=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/20c33d68-4eee-446f-a577-8846ad357e71
[1504 bytes], 000037.sst=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/09d29222-4e74-4f96-b386-fada894a5659
[978 bytes], 000031.sst=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/2dbec345-6e8a-4c4e-8e7e-a091a26450a0
[985 bytes], 000033.sst=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/92b35651-65f2-400f-bb0c-3a6985dbacc1
[971 bytes], 000035.sst=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/affd3ff7-a386-4792-949c-3a45046ad43f
[974 bytes], CURRENT=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/9e8e1d35-c776-4453-8ccf-2f293af4ab8d
[16 bytes], 000039.sst=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/fe7f7100-2f61-4b52-bf86-3d5c0a465cca
[968 bytes], OPTIONS-000029=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/85b3ccab-fc2d-4cc8-89ff-bc59468eb7cf
[46013 bytes], 000038.sst=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/dd46893b-9c5e-4848-80bb-06df0cec3b15
[971 bytes], 000030.sst=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/ed5b7433-701c-4985-84cd-d1fdb02746d0
[976 bytes], 000036.sst=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/ed0db297-3bf6-4665-8007-e72dc9cf96a0
[968 bytes], 000032.sst=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/8778f89c-af40-418b-b445-c5a4caac69aa
[975 bytes], 000034.sst=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/c5d7a495-2e4b-4c6e-80ab-02a06f28d730
[2097 bytes]}, metaStateHandle=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/c42cfb16-dafe-47cb-b746-8d7f1a2987b5
[8135 bytes], registered=true}}.
14072 [flink-akka.actor.default-dispatcher-8] DEBUG
org.apache.flink.runtime.state.SharedStateRegistryImpl [] - Scheduled delete of
state handle EmptyDiscardStateObject{b74fb79e-ffa0-4715-bd37-530e42c8295a}.
14072 [flink-akka.actor.default-dispatcher-8] DEBUG
org.apache.flink.runtime.state.SharedStateRegistryImpl [] - Scheduled delete of
state handle
Wrapped{IncrementalRemoteKeyedStateHandle{backendIdentifier=1323d441-6ee1-4550-9f5a-2c9371b18163,
stateHandleId=c9e952de-f7fc-4fb4-8542-47a416b1503f,
keyGroupRange=KeyGroupRange{startKeyGroup=3, endKeyGroup=3}, checkpointId=1,
sharedState={}, privateState={MANIFEST-000004=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/bd41577a-6ad5-4949-8aa6-a212e7cc5ee7
[1504 bytes], 000037.sst=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/a45ba92c-86cf-4b87-b821-f24062bc707a
[978 bytes], 000031.sst=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/1f5a6a50-f07a-43a4-97e0-4ff4bb4b7c5e
[985 bytes], 000033.sst=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/a7ac64ac-ba25-405e-9a71-b3dacb6be146
[971 bytes], 000035.sst=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/579219a0-f75a-4969-8a4e-f1fb42e96cd8
[974 bytes], CURRENT=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/5482073c-531c-4ba9-a9c6-969a30c5bee5
[16 bytes], 000039.sst=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/4b7ed710-9eca-4f47-80e1-96ea2646f084
[968 bytes], OPTIONS-000029=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/c1d4a8a8-3889-494d-96a7-443dbec20754
[46013 bytes], 000038.sst=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/b03df3a0-33bf-41cd-aefa-5483478164a7
[971 bytes], 000030.sst=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/51c777e6-973e-4630-9849-2e184b88596d
[976 bytes], 000036.sst=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/82f3464d-ebc5-4a20-8200-3769520dea00
[968 bytes], 000032.sst=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/55ac8a74-9a5c-4e8a-a7b6-c6fa8a9ade31
[975 bytes], 000034.sst=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/e091ef0b-fb7f-43be-ba46-5e9998f52a73
[2097 bytes]}, metaStateHandle=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/23720006-b2ca-438f-b7a5-95e9c7c02483
[8135 bytes], registered=true}}.
14072 [flink-akka.actor.default-dispatcher-8] DEBUG
org.apache.flink.runtime.state.SharedStateRegistryImpl [] - Scheduled delete of
state handle EmptyDiscardStateObject{fd48cd55-c1d3-4844-a976-572a80bde8e2}.
14072 [flink-akka.actor.default-dispatcher-8] DEBUG
org.apache.flink.runtime.state.SharedStateRegistryImpl [] - Scheduled delete of
state handle EmptyDiscardStateObject{7ed77adc-c7ea-4e59-a0b9-8745e5d4770a}.
14072 [flink-akka.actor.default-dispatcher-8] DEBUG
org.apache.flink.runtime.state.SharedStateRegistryImpl [] - Scheduled delete of
state handle
Wrapped{IncrementalRemoteKeyedStateHandle{backendIdentifier=c3e64345-2042-41d2-beba-02d354aa725b,
stateHandleId=74b12755-6624-4997-b545-f36df4c344d6,
keyGroupRange=KeyGroupRange{startKeyGroup=2, endKeyGroup=2}, checkpointId=1,
sharedState={}, privateState={MANIFEST-000004=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/a471413a-140a-41ab-a8d2-3f186eadc07a
[1581 bytes], 000037.sst=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/24c1a635-16d2-486a-9c0f-b5265b490c5b
[1002 bytes], 000031.sst=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/d6d69265-47e8-4616-b13c-e58a6a7f37fc
[1004 bytes], 000033.sst=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/c1f11a24-217d-4646-a4ab-a255d5ad3590
[995 bytes], 000035.sst=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/f6c8d992-876e-4159-8292-7cb571fbd975
[998 bytes], CURRENT=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/7ebe0727-a2d4-42da-a09d-0b2d6ad2b2d6
[16 bytes], 000039.sst=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/20e9d31a-1c31-4c96-90f0-93bd2bf1a5ce
[987 bytes], OPTIONS-000029=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/fb6137e9-2036-46e7-a9c1-e8fa284b9144
[46013 bytes], 000040.sst=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/41f9838b-0bdd-4a6c-ae2b-51544e522ac5
[977 bytes], 000038.sst=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/43124311-c3c2-4143-a117-7b7be285c08c
[2097 bytes], 000030.sst=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/698051e4-e19e-4d3a-9b56-26165b84bc6f
[999 bytes], 000036.sst=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/d0bc852a-8581-4a2e-969e-b1030afc5164
[982 bytes], 000032.sst=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/b9fba9c0-64a2-44ef-97c0-ad61f89d0b68
[991 bytes], 000034.sst=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/f0130bb7-d225-488e-9764-03bfbb62f78b
[2097 bytes]}, metaStateHandle=File State:
file:/tmp/junit3473564016309490181/checkpoints_b273b9b5-0c95-47eb-ba6d-b889b906a752/9680ae77c09d7de75d7fc0df6f0ffb08/taskowned/e97af735-7c1e-4ec9-92a8-411fbc7c6c87
[8135 bytes], registered=true}}.
{code}
We see the same outcome here as already described by [~Yanfei Lei] with these
missing artifacts causing the verification step to fail.
> Update reference data for Migration Tests
> -----------------------------------------
>
> Key: FLINK-31593
> URL: https://issues.apache.org/jira/browse/FLINK-31593
> Project: Flink
> Issue Type: Sub-task
> Reporter: Matthias Pohl
> Assignee: Matthias Pohl
> Priority: Major
> Labels: pull-request-available
> Attachments:
> FLINK-31593.StatefulJobSavepointMigrationITCase.create_snapshot.log,
> FLINK-31593.StatefulJobSavepointMigrationITCase.verify_snapshot.log
>
>
> # Update {{CURRENT_VERSION in TypeSerializerUpgradeTestBase}} with the new
> version. This will likely fail some tests because snapshots are missing for
> that version. Generate them, for example in
> {{TypeSerializerUpgradeTestBase.}}
> # (major/minor only) Update migration tests in master to cover migration
> from new version: (search for usages of FlinkV{{{}ersion{}}})
> ** AbstractOperatorRestoreTestBase
> ** CEPMigrationTest
> ** BucketingSinkMigrationTest
> ** FlinkKafkaConsumerBaseMigrationTest
> ** ContinuousFileProcessingMigrationTest
> ** WindowOperatorMigrationTest
> ** StatefulJobSavepointMigrationITCase
> ** StatefulJobWBroadcastStateMigrationITCase
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
