[
https://issues.apache.org/jira/browse/HDDS-8530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17720062#comment-17720062
]
Hemant Kumar commented on HDDS-8530:
------------------------------------
The problem is not lexicographical ordering in RocksDB table because on restore
we do sort the snapshots based on creation time:
[https://github.com/apache/ozone/blob/master/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SnapshotChainManager.java#L276]
On the deep dive, I found that it is problem with snapshot creation time itself
which is not unique across all OM nodes and causes that problem.
I wrote a test to reproduce it locally and based on logs you can see that
snapshots which caused the failure have messed up creation order.
Failure log:
{code:java}
java.io.IOException: Snapshot Chain corruption: previous snapshotID given but
no associated snapshot found in snapshot chain: SnapshotID
d55385d8-6583-4880-9ead-25cad63b390b
at
org.apache.hadoop.ozone.om.SnapshotChainManager.addSnapshotGlobal(SnapshotChainManager.java:86)
at
org.apache.hadoop.ozone.om.SnapshotChainManager.addSnapshot(SnapshotChainManager.java:292)
at
org.apache.hadoop.ozone.om.SnapshotChainManager.loadFromSnapshotInfoTable(SnapshotChainManager.java:282)
at
org.apache.hadoop.ozone.om.SnapshotChainManager.<init>(SnapshotChainManager.java:63)
at
org.apache.hadoop.ozone.om.OmMetadataManagerImpl.start(OmMetadataManagerImpl.java:517)
at
org.apache.hadoop.ozone.om.OmMetadataManagerImpl.<init>(OmMetadataManagerImpl.java:321)
at
org.apache.hadoop.ozone.om.OzoneManager.instantiateServices(OzoneManager.java:762)
at org.apache.hadoop.ozone.om.OzoneManager.restart(OzoneManager.java:1668)
at
org.apache.hadoop.ozone.MiniOzoneHAClusterImpl.restartOzoneManager(MiniOzoneHAClusterImpl.java:229)
at
org.apache.hadoop.ozone.om.TestOzoneManagerHA.resetCluster(TestOzoneManagerHA.java:202)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
{code}
Snapshot chain manager restore fails at snapshotId
*d55385d8-6583-4880-9ead-25cad63b390b* and probably either adding
*4b502e49-2e73-4ec7-a40e-9c440f4658e4* or
*2117ee6f-0005-48d8-9bfb-fa3422830427*.
{code:java}
created snapshot: 4b502e49-2e73-4ec7-a40e-9c440f4658e4 at created at timestamp:
1683236055206
created snapshot: 4b502e49-2e73-4ec7-a40e-9c440f4658e4 at created at timestamp:
1683236055209
created snapshot: 28e80ddc-beae-43b2-b44d-3118de803daf at created at timestamp:
1683236055264
created snapshot: d55385d8-6583-4880-9ead-25cad63b390b at created at timestamp:
1683236055299
created snapshot: 4b502e49-2e73-4ec7-a40e-9c440f4658e4 at created at timestamp:
1683236055395
created snapshot: 28e80ddc-beae-43b2-b44d-3118de803daf at created at timestamp:
1683236055396
created snapshot: d55385d8-6583-4880-9ead-25cad63b390b at created at timestamp:
1683236055396
created snapshot: 2117ee6f-0005-48d8-9bfb-fa3422830427 at created at timestamp:
1683236055397
created snapshot: 28e80ddc-beae-43b2-b44d-3118de803daf at created at timestamp:
1683236055449
created snapshot: d55385d8-6583-4880-9ead-25cad63b390b at created at timestamp:
1683236055449
created snapshot: 2117ee6f-0005-48d8-9bfb-fa3422830427 at created at timestamp:
1683236055450
created snapshot: 2117ee6f-0005-48d8-9bfb-fa3422830427 at created at timestamp:
1683236055493
{code}
Full snapshot Creation log. You can see before snapshotId:
*d55385d8-6583-4880-9ead-25cad63b390b*, all three OM create snapshot in order.
{code:java}
created snapshot: f7c5951a-ba18-47d3-9798-13933d606e63 at created at timestamp:
1683236051818
created snapshot: f7c5951a-ba18-47d3-9798-13933d606e63 at created at timestamp:
1683236051817
created snapshot: f7c5951a-ba18-47d3-9798-13933d606e63 at created at timestamp:
1683236051818
created snapshot: b978f33e-0aa1-4756-94f5-ad3d3ae57e66 at created at timestamp:
1683236052049
created snapshot: b978f33e-0aa1-4756-94f5-ad3d3ae57e66 at created at timestamp:
1683236052056
created snapshot: b978f33e-0aa1-4756-94f5-ad3d3ae57e66 at created at timestamp:
1683236052056
created snapshot: 497f6e8f-36b4-4116-b4cf-8ee41c528b3f at created at timestamp:
1683236052385
created snapshot: 497f6e8f-36b4-4116-b4cf-8ee41c528b3f at created at timestamp:
1683236052388
created snapshot: 497f6e8f-36b4-4116-b4cf-8ee41c528b3f at created at timestamp:
1683236052428
created snapshot: 90a6daf4-005a-4c23-a130-adbcb9390a12 at created at timestamp:
1683236052491
created snapshot: 90a6daf4-005a-4c23-a130-adbcb9390a12 at created at timestamp:
1683236052499
created snapshot: 90a6daf4-005a-4c23-a130-adbcb9390a12 at created at timestamp:
1683236052506
created snapshot: b3b4cce7-df5c-4e81-9fd5-adfc2bfb1099 at created at timestamp:
1683236052638
created snapshot: b3b4cce7-df5c-4e81-9fd5-adfc2bfb1099 at created at timestamp:
1683236052646
created snapshot: b3b4cce7-df5c-4e81-9fd5-adfc2bfb1099 at created at timestamp:
1683236052649
created snapshot: edf7b277-26f5-482b-83c4-ee23414a44dc at created at timestamp:
1683236052685
created snapshot: edf7b277-26f5-482b-83c4-ee23414a44dc at created at timestamp:
1683236052693
created snapshot: edf7b277-26f5-482b-83c4-ee23414a44dc at created at timestamp:
1683236052693
created snapshot: 618165ce-7b6d-4058-b768-c583e37a2aa3 at created at timestamp:
1683236052770
created snapshot: 618165ce-7b6d-4058-b768-c583e37a2aa3 at created at timestamp:
1683236052772
created snapshot: 618165ce-7b6d-4058-b768-c583e37a2aa3 at created at timestamp:
1683236052772
created snapshot: 1e41a22a-3167-42cf-8f1e-0fe23286b00b at created at timestamp:
1683236052852
created snapshot: 1e41a22a-3167-42cf-8f1e-0fe23286b00b at created at timestamp:
1683236052857
created snapshot: 1e41a22a-3167-42cf-8f1e-0fe23286b00b at created at timestamp:
1683236052857
created snapshot: 683b2059-b91a-4cbd-8f51-239c88a9364c at created at timestamp:
1683236052902
created snapshot: 683b2059-b91a-4cbd-8f51-239c88a9364c at created at timestamp:
1683236052908
created snapshot: 683b2059-b91a-4cbd-8f51-239c88a9364c at created at timestamp:
1683236052908
created snapshot: 7886702f-31d8-484b-ab35-b87823a8c5ab at created at timestamp:
1683236052940
created snapshot: 7886702f-31d8-484b-ab35-b87823a8c5ab at created at timestamp:
1683236052943
created snapshot: 7886702f-31d8-484b-ab35-b87823a8c5ab at created at timestamp:
1683236052945
created snapshot: e4da929f-e43a-4c95-8d1c-6ad17e6b10cb at created at timestamp:
1683236052995
created snapshot: e4da929f-e43a-4c95-8d1c-6ad17e6b10cb at created at timestamp:
1683236053000
created snapshot: e4da929f-e43a-4c95-8d1c-6ad17e6b10cb at created at timestamp:
1683236053000
created snapshot: 2105c507-18b7-4d97-aeff-21c99311d30b at created at timestamp:
1683236053124
created snapshot: 2105c507-18b7-4d97-aeff-21c99311d30b at created at timestamp:
1683236053124
created snapshot: 2105c507-18b7-4d97-aeff-21c99311d30b at created at timestamp:
1683236053123
created snapshot: 95d7ea9a-af36-4196-893d-65fc842a7c53 at created at timestamp:
1683236053216
created snapshot: 95d7ea9a-af36-4196-893d-65fc842a7c53 at created at timestamp:
1683236053221
created snapshot: 95d7ea9a-af36-4196-893d-65fc842a7c53 at created at timestamp:
1683236053221
created snapshot: d440c3eb-0f7b-4e0d-9d44-3f25b2c30d7e at created at timestamp:
1683236053258
created snapshot: d440c3eb-0f7b-4e0d-9d44-3f25b2c30d7e at created at timestamp:
1683236053264
created snapshot: d440c3eb-0f7b-4e0d-9d44-3f25b2c30d7e at created at timestamp:
1683236053264
created snapshot: 89445e82-4e3e-4eb2-9cdc-94cfc15bb2aa at created at timestamp:
1683236053302
created snapshot: 89445e82-4e3e-4eb2-9cdc-94cfc15bb2aa at created at timestamp:
1683236053308
created snapshot: 89445e82-4e3e-4eb2-9cdc-94cfc15bb2aa at created at timestamp:
1683236053308
created snapshot: d7734528-5490-4996-b9f0-1a721fe1b34e at created at timestamp:
1683236053342
created snapshot: d7734528-5490-4996-b9f0-1a721fe1b34e at created at timestamp:
1683236053347
created snapshot: d7734528-5490-4996-b9f0-1a721fe1b34e at created at timestamp:
1683236053347
created snapshot: 7471273d-6fc1-4b12-8ad4-2283ef40f003 at created at timestamp:
1683236053369
created snapshot: 7471273d-6fc1-4b12-8ad4-2283ef40f003 at created at timestamp:
1683236053375
created snapshot: 7471273d-6fc1-4b12-8ad4-2283ef40f003 at created at timestamp:
1683236053375
created snapshot: 17317d19-e60b-4074-8aa3-53617a783a4a at created at timestamp:
1683236053414
created snapshot: 17317d19-e60b-4074-8aa3-53617a783a4a at created at timestamp:
1683236053420
created snapshot: 17317d19-e60b-4074-8aa3-53617a783a4a at created at timestamp:
1683236053420
created snapshot: b4e97e31-c09c-4012-8f6f-877166955612 at created at timestamp:
1683236053453
created snapshot: b4e97e31-c09c-4012-8f6f-877166955612 at created at timestamp:
1683236053455
created snapshot: b4e97e31-c09c-4012-8f6f-877166955612 at created at timestamp:
1683236053455
created snapshot: 48f4e9d3-d40c-4769-978c-fbb58bcbd52c at created at timestamp:
1683236053491
created snapshot: 48f4e9d3-d40c-4769-978c-fbb58bcbd52c at created at timestamp:
1683236053494
created snapshot: 48f4e9d3-d40c-4769-978c-fbb58bcbd52c at created at timestamp:
1683236053494
created snapshot: 4094e343-ba2d-48ed-a509-c03f64cc9c76 at created at timestamp:
1683236053534
created snapshot: 4094e343-ba2d-48ed-a509-c03f64cc9c76 at created at timestamp:
1683236053536
created snapshot: 4094e343-ba2d-48ed-a509-c03f64cc9c76 at created at timestamp:
1683236053537
created snapshot: 99bdc23b-dac9-4a1f-8e14-83f633d22dfa at created at timestamp:
1683236053574
created snapshot: 99bdc23b-dac9-4a1f-8e14-83f633d22dfa at created at timestamp:
1683236053579
created snapshot: 99bdc23b-dac9-4a1f-8e14-83f633d22dfa at created at timestamp:
1683236053579
created snapshot: 1dfd04af-01d3-4f65-ad55-1aaa6e339110 at created at timestamp:
1683236053610
created snapshot: 1dfd04af-01d3-4f65-ad55-1aaa6e339110 at created at timestamp:
1683236053615
created snapshot: 1dfd04af-01d3-4f65-ad55-1aaa6e339110 at created at timestamp:
1683236053615
created snapshot: 696d4f10-1802-4e3c-81ce-9a29abb83bec at created at timestamp:
1683236053644
created snapshot: 696d4f10-1802-4e3c-81ce-9a29abb83bec at created at timestamp:
1683236053650
created snapshot: 696d4f10-1802-4e3c-81ce-9a29abb83bec at created at timestamp:
1683236053650
created snapshot: 23170bca-8367-415c-8d4e-b53bfbf1a065 at created at timestamp:
1683236053684
created snapshot: 23170bca-8367-415c-8d4e-b53bfbf1a065 at created at timestamp:
1683236053691
created snapshot: 23170bca-8367-415c-8d4e-b53bfbf1a065 at created at timestamp:
1683236053691
created snapshot: d094c19a-e523-454d-a618-0736a2de04ea at created at timestamp:
1683236053715
created snapshot: d094c19a-e523-454d-a618-0736a2de04ea at created at timestamp:
1683236053718
created snapshot: d094c19a-e523-454d-a618-0736a2de04ea at created at timestamp:
1683236053718
created snapshot: 4c78a8ac-c124-4864-ac1d-ea3ef4c3bd25 at created at timestamp:
1683236053740
created snapshot: 4c78a8ac-c124-4864-ac1d-ea3ef4c3bd25 at created at timestamp:
1683236053746
created snapshot: 4c78a8ac-c124-4864-ac1d-ea3ef4c3bd25 at created at timestamp:
1683236053746
created snapshot: 3ff71eb9-9169-4013-b37a-bed0956cc0f5 at created at timestamp:
1683236053766
created snapshot: 3ff71eb9-9169-4013-b37a-bed0956cc0f5 at created at timestamp:
1683236053767
created snapshot: 3ff71eb9-9169-4013-b37a-bed0956cc0f5 at created at timestamp:
1683236053767
created snapshot: 3378114c-8b16-4f84-b6a1-cfbfd2cfe386 at created at timestamp:
1683236053797
created snapshot: 3378114c-8b16-4f84-b6a1-cfbfd2cfe386 at created at timestamp:
1683236053800
created snapshot: 3378114c-8b16-4f84-b6a1-cfbfd2cfe386 at created at timestamp:
1683236053800
created snapshot: 45f27a60-5802-4e08-987a-c20ac2a7193a at created at timestamp:
1683236053824
created snapshot: 45f27a60-5802-4e08-987a-c20ac2a7193a at created at timestamp:
1683236053825
created snapshot: 45f27a60-5802-4e08-987a-c20ac2a7193a at created at timestamp:
1683236053832
created snapshot: 8001850a-ff6b-4b4a-8099-6223ae7e63a6 at created at timestamp:
1683236053865
created snapshot: 8001850a-ff6b-4b4a-8099-6223ae7e63a6 at created at timestamp:
1683236053869
created snapshot: 8001850a-ff6b-4b4a-8099-6223ae7e63a6 at created at timestamp:
1683236053869
created snapshot: d140c4bb-1b37-427f-8ca8-85f7cc2744e6 at created at timestamp:
1683236053907
created snapshot: d140c4bb-1b37-427f-8ca8-85f7cc2744e6 at created at timestamp:
1683236053911
created snapshot: d140c4bb-1b37-427f-8ca8-85f7cc2744e6 at created at timestamp:
1683236053911
created snapshot: e654125b-f15c-4fa5-8b96-9e4127b1a79f at created at timestamp:
1683236053970
created snapshot: e654125b-f15c-4fa5-8b96-9e4127b1a79f at created at timestamp:
1683236053970
created snapshot: e654125b-f15c-4fa5-8b96-9e4127b1a79f at created at timestamp:
1683236053970
created snapshot: 3b14010a-7d00-4b37-9ab8-f19cfe3e1065 at created at timestamp:
1683236054037
created snapshot: 3b14010a-7d00-4b37-9ab8-f19cfe3e1065 at created at timestamp:
1683236054039
created snapshot: 3b14010a-7d00-4b37-9ab8-f19cfe3e1065 at created at timestamp:
1683236054040
created snapshot: 72841901-68bd-4e4a-9a0d-f30e536e81fa at created at timestamp:
1683236054145
created snapshot: 72841901-68bd-4e4a-9a0d-f30e536e81fa at created at timestamp:
1683236054148
created snapshot: 72841901-68bd-4e4a-9a0d-f30e536e81fa at created at timestamp:
1683236054148
created snapshot: 735e5518-24b6-466e-9b6d-34d75dc0a110 at created at timestamp:
1683236054214
created snapshot: 735e5518-24b6-466e-9b6d-34d75dc0a110 at created at timestamp:
1683236054221
created snapshot: 735e5518-24b6-466e-9b6d-34d75dc0a110 at created at timestamp:
1683236054221
created snapshot: 57b884f1-a235-47f1-9383-623dbfdab34e at created at timestamp:
1683236054292
created snapshot: 57b884f1-a235-47f1-9383-623dbfdab34e at created at timestamp:
1683236054295
created snapshot: 57b884f1-a235-47f1-9383-623dbfdab34e at created at timestamp:
1683236054296
created snapshot: 48694ca3-4672-4fcb-a05a-13e773ce9afa at created at timestamp:
1683236054372
created snapshot: 48694ca3-4672-4fcb-a05a-13e773ce9afa at created at timestamp:
1683236054375
created snapshot: 48694ca3-4672-4fcb-a05a-13e773ce9afa at created at timestamp:
1683236054375
created snapshot: 1c49fb62-539e-405b-ae6c-0a05abe882e7 at created at timestamp:
1683236054455
created snapshot: 1c49fb62-539e-405b-ae6c-0a05abe882e7 at created at timestamp:
1683236054457
created snapshot: 1c49fb62-539e-405b-ae6c-0a05abe882e7 at created at timestamp:
1683236054458
created snapshot: c90e222d-ee60-48db-a4c4-d48b6a23d01c at created at timestamp:
1683236054515
created snapshot: c90e222d-ee60-48db-a4c4-d48b6a23d01c at created at timestamp:
1683236054518
created snapshot: c90e222d-ee60-48db-a4c4-d48b6a23d01c at created at timestamp:
1683236054518
created snapshot: 16c8a372-cc81-4fe7-a257-a11fa233883e at created at timestamp:
1683236054557
created snapshot: 16c8a372-cc81-4fe7-a257-a11fa233883e at created at timestamp:
1683236054559
created snapshot: 16c8a372-cc81-4fe7-a257-a11fa233883e at created at timestamp:
1683236054560
created snapshot: 3489804c-dced-4236-8f19-28757f6f2008 at created at timestamp:
1683236054587
created snapshot: 3489804c-dced-4236-8f19-28757f6f2008 at created at timestamp:
1683236054589
created snapshot: 3489804c-dced-4236-8f19-28757f6f2008 at created at timestamp:
1683236054589
created snapshot: 1677cef2-f91c-4f8b-893e-1db83051c7f1 at created at timestamp:
1683236054625
created snapshot: 1677cef2-f91c-4f8b-893e-1db83051c7f1 at created at timestamp:
1683236054627
created snapshot: 1677cef2-f91c-4f8b-893e-1db83051c7f1 at created at timestamp:
1683236054628
created snapshot: 56a2aed5-e6cd-41c0-8fc1-c899509f78c9 at created at timestamp:
1683236054689
created snapshot: 56a2aed5-e6cd-41c0-8fc1-c899509f78c9 at created at timestamp:
1683236054695
created snapshot: 56a2aed5-e6cd-41c0-8fc1-c899509f78c9 at created at timestamp:
1683236054695
created snapshot: 1209ea3b-70c7-43d6-a9c3-5a9b7daf26c8 at created at timestamp:
1683236054733
created snapshot: 1209ea3b-70c7-43d6-a9c3-5a9b7daf26c8 at created at timestamp:
1683236054740
created snapshot: 1209ea3b-70c7-43d6-a9c3-5a9b7daf26c8 at created at timestamp:
1683236054740
created snapshot: 84b30a8a-3485-4ced-8042-c01fab072317 at created at timestamp:
1683236054774
created snapshot: 84b30a8a-3485-4ced-8042-c01fab072317 at created at timestamp:
1683236054774
created snapshot: 84b30a8a-3485-4ced-8042-c01fab072317 at created at timestamp:
1683236054774
created snapshot: b21d4c6d-183f-4430-812e-cbd2d8170852 at created at timestamp:
1683236054801
created snapshot: b21d4c6d-183f-4430-812e-cbd2d8170852 at created at timestamp:
1683236054801
created snapshot: b21d4c6d-183f-4430-812e-cbd2d8170852 at created at timestamp:
1683236054801
created snapshot: 20a50a61-0192-4b64-a8a8-e32ee404e946 at created at timestamp:
1683236054845
created snapshot: 20a50a61-0192-4b64-a8a8-e32ee404e946 at created at timestamp:
1683236054850
created snapshot: 20a50a61-0192-4b64-a8a8-e32ee404e946 at created at timestamp:
1683236054850
created snapshot: a77c1047-cde2-42da-bb3d-c47c54bf3958 at created at timestamp:
1683236054885
created snapshot: a77c1047-cde2-42da-bb3d-c47c54bf3958 at created at timestamp:
1683236054889
created snapshot: a77c1047-cde2-42da-bb3d-c47c54bf3958 at created at timestamp:
1683236054889
created snapshot: a0a420f4-9c21-4655-99fb-38cdf28c4c6c at created at timestamp:
1683236054923
created snapshot: a0a420f4-9c21-4655-99fb-38cdf28c4c6c at created at timestamp:
1683236054926
created snapshot: a0a420f4-9c21-4655-99fb-38cdf28c4c6c at created at timestamp:
1683236054926
created snapshot: 35b9f50e-2200-44b9-9f8d-231c9e7cb74f at created at timestamp:
1683236054965
created snapshot: 35b9f50e-2200-44b9-9f8d-231c9e7cb74f at created at timestamp:
1683236054975
created snapshot: 35b9f50e-2200-44b9-9f8d-231c9e7cb74f at created at timestamp:
1683236054975
created snapshot: 448941e6-68f2-461d-be4d-4b6f4bb4082e at created at timestamp:
1683236055016
created snapshot: 448941e6-68f2-461d-be4d-4b6f4bb4082e at created at timestamp:
1683236055022
created snapshot: 448941e6-68f2-461d-be4d-4b6f4bb4082e at created at timestamp:
1683236055022
created snapshot: 39602ef6-4db2-4410-af4e-433b8b62e49d at created at timestamp:
1683236055053
created snapshot: 39602ef6-4db2-4410-af4e-433b8b62e49d at created at timestamp:
1683236055058
created snapshot: 39602ef6-4db2-4410-af4e-433b8b62e49d at created at timestamp:
1683236055058
created snapshot: 8a1eba7d-d9dd-4f87-a106-29e8a4502eba at created at timestamp:
1683236055094
created snapshot: 8a1eba7d-d9dd-4f87-a106-29e8a4502eba at created at timestamp:
1683236055098
created snapshot: 8a1eba7d-d9dd-4f87-a106-29e8a4502eba at created at timestamp:
1683236055103
created snapshot: 0814d2f5-f07b-455e-957d-4d966c05545a at created at timestamp:
1683236055125
created snapshot: 0814d2f5-f07b-455e-957d-4d966c05545a at created at timestamp:
1683236055125
created snapshot: 0814d2f5-f07b-455e-957d-4d966c05545a at created at timestamp:
1683236055125
created snapshot: 1fa28d10-24fb-4377-bc4c-128605f0e179 at created at timestamp:
1683236055148
created snapshot: 1fa28d10-24fb-4377-bc4c-128605f0e179 at created at timestamp:
1683236055150
created snapshot: 1fa28d10-24fb-4377-bc4c-128605f0e179 at created at timestamp:
1683236055150
created snapshot: 4b502e49-2e73-4ec7-a40e-9c440f4658e4 at created at timestamp:
1683236055206
created snapshot: 4b502e49-2e73-4ec7-a40e-9c440f4658e4 at created at timestamp:
1683236055209
created snapshot: 28e80ddc-beae-43b2-b44d-3118de803daf at created at timestamp:
1683236055264
created snapshot: d55385d8-6583-4880-9ead-25cad63b390b at created at timestamp:
1683236055299
created snapshot: 4b502e49-2e73-4ec7-a40e-9c440f4658e4 at created at timestamp:
1683236055395
created snapshot: 28e80ddc-beae-43b2-b44d-3118de803daf at created at timestamp:
1683236055396
created snapshot: d55385d8-6583-4880-9ead-25cad63b390b at created at timestamp:
1683236055396
created snapshot: 2117ee6f-0005-48d8-9bfb-fa3422830427 at created at timestamp:
1683236055397
created snapshot: 28e80ddc-beae-43b2-b44d-3118de803daf at created at timestamp:
1683236055449
created snapshot: d55385d8-6583-4880-9ead-25cad63b390b at created at timestamp:
1683236055449
created snapshot: 2117ee6f-0005-48d8-9bfb-fa3422830427 at created at timestamp:
1683236055450
created snapshot: 2117ee6f-0005-48d8-9bfb-fa3422830427 at created at timestamp:
1683236055493
created snapshot: abd07d91-032c-472a-aa87-85b0d379038d at created at timestamp:
1683236055515
created snapshot: abd07d91-032c-472a-aa87-85b0d379038d at created at timestamp:
1683236055520
created snapshot: abd07d91-032c-472a-aa87-85b0d379038d at created at timestamp:
1683236055520
created snapshot: a501024b-662d-48dc-83aa-8edacd43afd6 at created at timestamp:
1683236055547
created snapshot: a501024b-662d-48dc-83aa-8edacd43afd6 at created at timestamp:
1683236055547
created snapshot: a501024b-662d-48dc-83aa-8edacd43afd6 at created at timestamp:
1683236055547
created snapshot: de0316e7-3779-4ecc-baca-5a9c95951e3d at created at timestamp:
1683236055591
created snapshot: de0316e7-3779-4ecc-baca-5a9c95951e3d at created at timestamp:
1683236055598
created snapshot: de0316e7-3779-4ecc-baca-5a9c95951e3d at created at timestamp:
1683236055598
created snapshot: 7e08c8f8-6cc9-4292-b031-61ba522ede87 at created at timestamp:
1683236055644
created snapshot: 7e08c8f8-6cc9-4292-b031-61ba522ede87 at created at timestamp:
1683236055647
created snapshot: 7e08c8f8-6cc9-4292-b031-61ba522ede87 at created at timestamp:
1683236055647
created snapshot: f501ced6-7017-4f91-8f54-bd160b8c4603 at created at timestamp:
1683236055694
created snapshot: f501ced6-7017-4f91-8f54-bd160b8c4603 at created at timestamp:
1683236055699
created snapshot: f501ced6-7017-4f91-8f54-bd160b8c4603 at created at timestamp:
1683236055699
created snapshot: c43deb1f-b472-4f56-a608-9b4c470e205a at created at timestamp:
1683236055728
created snapshot: c43deb1f-b472-4f56-a608-9b4c470e205a at created at timestamp:
1683236055733
created snapshot: c43deb1f-b472-4f56-a608-9b4c470e205a at created at timestamp:
1683236055733
created snapshot: 1678824a-a1fe-4872-aa29-1168dcf42089 at created at timestamp:
1683236055790
created snapshot: 1678824a-a1fe-4872-aa29-1168dcf42089 at created at timestamp:
1683236055796
created snapshot: 1678824a-a1fe-4872-aa29-1168dcf42089 at created at timestamp:
1683236055796
created snapshot: ed3d2858-3321-4efd-aced-0ca405c86314 at created at timestamp:
1683236055832
created snapshot: ed3d2858-3321-4efd-aced-0ca405c86314 at created at timestamp:
1683236055839
created snapshot: ed3d2858-3321-4efd-aced-0ca405c86314 at created at timestamp:
1683236055839
created snapshot: 2c299784-0e97-4402-b22e-d3f5e1ca6baf at created at timestamp:
1683236055870
created snapshot: 2c299784-0e97-4402-b22e-d3f5e1ca6baf at created at timestamp:
1683236055872
created snapshot: 2c299784-0e97-4402-b22e-d3f5e1ca6baf at created at timestamp:
1683236055872
created snapshot: 9e600cb5-e962-4053-a41b-a8e5e3bb9405 at created at timestamp:
1683236055899
created snapshot: 9e600cb5-e962-4053-a41b-a8e5e3bb9405 at created at timestamp:
1683236055899
created snapshot: 9e600cb5-e962-4053-a41b-a8e5e3bb9405 at created at timestamp:
1683236055900
created snapshot: 7e16d7b0-d5bc-4e60-8d4a-f625eca5552f at created at timestamp:
1683236055923
created snapshot: 7e16d7b0-d5bc-4e60-8d4a-f625eca5552f at created at timestamp:
1683236055924
created snapshot: 7e16d7b0-d5bc-4e60-8d4a-f625eca5552f at created at timestamp:
1683236055924
created snapshot: 3a441be2-fe8d-4c28-9723-969db20ef277 at created at timestamp:
1683236055972
created snapshot: 3a441be2-fe8d-4c28-9723-969db20ef277 at created at timestamp:
1683236055977
created snapshot: 3a441be2-fe8d-4c28-9723-969db20ef277 at created at timestamp:
1683236055977
created snapshot: 09dbe437-71a3-4f11-9d18-a194d317b811 at created at timestamp:
1683236056027
created snapshot: 09dbe437-71a3-4f11-9d18-a194d317b811 at created at timestamp:
1683236056032
created snapshot: 09dbe437-71a3-4f11-9d18-a194d317b811 at created at timestamp:
1683236056032
created snapshot: 14f20d1a-5868-469d-ab50-ca4cd899604b at created at timestamp:
1683236056078
created snapshot: 14f20d1a-5868-469d-ab50-ca4cd899604b at created at timestamp:
1683236056082
created snapshot: 14f20d1a-5868-469d-ab50-ca4cd899604b at created at timestamp:
1683236056082
created snapshot: 6c3090de-962d-41cd-afa5-d7ec7d52de03 at created at timestamp:
1683236056117
created snapshot: 6c3090de-962d-41cd-afa5-d7ec7d52de03 at created at timestamp:
1683236056119
created snapshot: 6c3090de-962d-41cd-afa5-d7ec7d52de03 at created at timestamp:
1683236056119
created snapshot: 943c382b-55df-46ff-a58f-455b922839ae at created at timestamp:
1683236056152
created snapshot: 943c382b-55df-46ff-a58f-455b922839ae at created at timestamp:
1683236056158
created snapshot: 943c382b-55df-46ff-a58f-455b922839ae at created at timestamp:
1683236056158
created snapshot: ad3e400a-31b2-4c98-922e-dbdb4b2f3e66 at created at timestamp:
1683236056190
created snapshot: ad3e400a-31b2-4c98-922e-dbdb4b2f3e66 at created at timestamp:
1683236056193
created snapshot: ad3e400a-31b2-4c98-922e-dbdb4b2f3e66 at created at timestamp:
1683236056193
created snapshot: ceecbfde-aaae-41e4-8cf1-ebf2b99a2a2c at created at timestamp:
1683236056223
created snapshot: ceecbfde-aaae-41e4-8cf1-ebf2b99a2a2c at created at timestamp:
1683236056235
created snapshot: ceecbfde-aaae-41e4-8cf1-ebf2b99a2a2c at created at timestamp:
1683236056235
created snapshot: d9021374-6107-4969-827a-62e8bda0e347 at created at timestamp:
1683236056276
created snapshot: d9021374-6107-4969-827a-62e8bda0e347 at created at timestamp:
1683236056276
created snapshot: d9021374-6107-4969-827a-62e8bda0e347 at created at timestamp:
1683236056276
created snapshot: bb3594b8-a1c3-438a-9cf1-3b2a610e3b6a at created at timestamp:
1683236056292
created snapshot: bb3594b8-a1c3-438a-9cf1-3b2a610e3b6a at created at timestamp:
1683236056292
created snapshot: bb3594b8-a1c3-438a-9cf1-3b2a610e3b6a at created at timestamp:
1683236056292
created snapshot: 16ccd5f8-4a62-4fb8-b046-85d1a33b07df at created at timestamp:
1683236056324
created snapshot: 16ccd5f8-4a62-4fb8-b046-85d1a33b07df at created at timestamp:
1683236056328
created snapshot: 16ccd5f8-4a62-4fb8-b046-85d1a33b07df at created at timestamp:
1683236056328
created snapshot: 0eaa87f4-9975-4e95-8558-1a60379cc9a9 at created at timestamp:
1683236056368
created snapshot: 0eaa87f4-9975-4e95-8558-1a60379cc9a9 at created at timestamp:
1683236056373
created snapshot: 0eaa87f4-9975-4e95-8558-1a60379cc9a9 at created at timestamp:
1683236056373
created snapshot: 9ee92859-01f9-485c-aee7-83d0f59fc852 at created at timestamp:
1683236056406
created snapshot: 9ee92859-01f9-485c-aee7-83d0f59fc852 at created at timestamp:
1683236056406
created snapshot: 9ee92859-01f9-485c-aee7-83d0f59fc852 at created at timestamp:
1683236056406
created snapshot: e9ba8dd2-2942-4fcf-937c-c2cd54194a40 at created at timestamp:
1683236056442
created snapshot: e9ba8dd2-2942-4fcf-937c-c2cd54194a40 at created at timestamp:
1683236056446
created snapshot: e9ba8dd2-2942-4fcf-937c-c2cd54194a40 at created at timestamp:
1683236056446
created snapshot: 2d45d300-fabf-4557-8cac-55e8416d3f86 at created at timestamp:
1683236056477
created snapshot: 2d45d300-fabf-4557-8cac-55e8416d3f86 at created at timestamp:
1683236056480
created snapshot: 2d45d300-fabf-4557-8cac-55e8416d3f86 at created at timestamp:
1683236056480
created snapshot: 60b27be6-77fc-4932-892d-a0a3571f2b8e at created at timestamp:
1683236056525
created snapshot: 60b27be6-77fc-4932-892d-a0a3571f2b8e at created at timestamp:
1683236056528
created snapshot: 60b27be6-77fc-4932-892d-a0a3571f2b8e at created at timestamp:
1683236056528
created snapshot: 230f9c33-90ff-48d9-a899-b4e6c445e59e at created at timestamp:
1683236056569
created snapshot: 230f9c33-90ff-48d9-a899-b4e6c445e59e at created at timestamp:
1683236056575
created snapshot: 230f9c33-90ff-48d9-a899-b4e6c445e59e at created at timestamp:
1683236056575
created snapshot: be70820e-79aa-48de-95ad-5ee7a99773e0 at created at timestamp:
1683236056607
created snapshot: be70820e-79aa-48de-95ad-5ee7a99773e0 at created at timestamp:
1683236056612
created snapshot: be70820e-79aa-48de-95ad-5ee7a99773e0 at created at timestamp:
1683236056616
created snapshot: f70af9d3-09a8-47b2-a23a-73b51aa326fa at created at timestamp:
1683236056652
created snapshot: f70af9d3-09a8-47b2-a23a-73b51aa326fa at created at timestamp:
1683236056654
created snapshot: f70af9d3-09a8-47b2-a23a-73b51aa326fa at created at timestamp:
1683236056654
created snapshot: 889fb02e-ef55-4ec0-8d33-c1e63a27aca4 at created at timestamp:
1683236056669
created snapshot: 889fb02e-ef55-4ec0-8d33-c1e63a27aca4 at created at timestamp:
1683236056669
created snapshot: 889fb02e-ef55-4ec0-8d33-c1e63a27aca4 at created at timestamp:
1683236056669
created snapshot: 937f394b-8a31-4d7c-a7d4-47322ca48abc at created at timestamp:
1683236056700
created snapshot: 937f394b-8a31-4d7c-a7d4-47322ca48abc at created at timestamp:
1683236056701
created snapshot: 937f394b-8a31-4d7c-a7d4-47322ca48abc at created at timestamp:
1683236056701
created snapshot: ea3a3867-bb4b-473d-befd-1ac2c8dabdc9 at created at timestamp:
1683236056749
created snapshot: ea3a3867-bb4b-473d-befd-1ac2c8dabdc9 at created at timestamp:
1683236056749
created snapshot: ea3a3867-bb4b-473d-befd-1ac2c8dabdc9 at created at timestamp:
1683236056750
created snapshot: 522bc227-16cd-4823-8506-6f6795f2c8e9 at created at timestamp:
1683236056791
created snapshot: 522bc227-16cd-4823-8506-6f6795f2c8e9 at created at timestamp:
1683236056791
created snapshot: 522bc227-16cd-4823-8506-6f6795f2c8e9 at created at timestamp:
1683236056791
created snapshot: e11862ec-b8a5-4fb2-b9fb-a863756be6b3 at created at timestamp:
1683236056829
created snapshot: e11862ec-b8a5-4fb2-b9fb-a863756be6b3 at created at timestamp:
1683236056833
created snapshot: e11862ec-b8a5-4fb2-b9fb-a863756be6b3 at created at timestamp:
1683236056833
created snapshot: 417311c9-a521-4507-9bd1-0b777005c669 at created at timestamp:
1683236056862
created snapshot: 417311c9-a521-4507-9bd1-0b777005c669 at created at timestamp:
1683236056868
created snapshot: 417311c9-a521-4507-9bd1-0b777005c669 at created at timestamp:
1683236056868
created snapshot: 084139d6-fda1-42ce-9125-b96f6bac44fc at created at timestamp:
1683236056900
created snapshot: 084139d6-fda1-42ce-9125-b96f6bac44fc at created at timestamp:
1683236056906
created snapshot: 084139d6-fda1-42ce-9125-b96f6bac44fc at created at timestamp:
1683236056906
created snapshot: 0461b831-4227-462f-940b-a2e729150eba at created at timestamp:
1683236056933
created snapshot: 0461b831-4227-462f-940b-a2e729150eba at created at timestamp:
1683236056938
created snapshot: 0461b831-4227-462f-940b-a2e729150eba at created at timestamp:
1683236056943
created snapshot: 4f0674ad-d320-400f-b253-8273ee64a72a at created at timestamp:
1683236056999
created snapshot: 4f0674ad-d320-400f-b253-8273ee64a72a at created at timestamp:
1683236057006
created snapshot: 4f0674ad-d320-400f-b253-8273ee64a72a at created at timestamp:
1683236057006
created snapshot: d98a05ea-809b-46b9-95fe-af205db1ce19 at created at timestamp:
1683236057061
created snapshot: d98a05ea-809b-46b9-95fe-af205db1ce19 at created at timestamp:
1683236057067
created snapshot: d98a05ea-809b-46b9-95fe-af205db1ce19 at created at timestamp:
1683236057067
created snapshot: 09b57643-7f3e-4602-ba2e-e00a0b3a5940 at created at timestamp:
1683236057094
created snapshot: 09b57643-7f3e-4602-ba2e-e00a0b3a5940 at created at timestamp:
1683236057094
created snapshot: 09b57643-7f3e-4602-ba2e-e00a0b3a5940 at created at timestamp:
1683236057094
{code}
> [snapshot] OM crash on restart due to Snapshot Chain corruption
> ---------------------------------------------------------------
>
> Key: HDDS-8530
> URL: https://issues.apache.org/jira/browse/HDDS-8530
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Manager
> Reporter: Jyotirmoy Sinha
> Priority: Major
> Labels: ozone-snapshot
> Attachments: Screenshot 2023-05-05 at 5.20.22 PM.png, Screenshot
> 2023-05-05 at 5.51.26 PM.png
>
>
> snapshotTable is sorted lexicographically and assumption that previous
> snapshot always exist is wrong
> OM error stacktrace -
> {code:java}
> 2023-05-03 18:59:48,889 [main] INFO
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer: Can't find SST '032458'
> 2023-05-03 18:59:48,889 [main] INFO
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer: Can't find SST '032459'
> 2023-05-03 18:59:48,890 [main] INFO
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer: Can't find SST '032466'
> 2023-05-03 18:59:48,890 [main] INFO
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer: Can't find SST '032475'
> 2023-05-03 18:59:48,890 [main] INFO
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer: Can't find SST '032480'
> 2023-05-03 18:59:48,890 [main] INFO
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer: Can't find SST '032482'
> 2023-05-03 18:59:48,891 [main] INFO
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer: Can't find SST '032491'
> 2023-05-03 18:59:48,891 [main] INFO
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer: Can't find SST '032495'
> 2023-05-03 18:59:48,891 [main] INFO
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer: Can't find SST '032497'
> 2023-05-03 18:59:48,891 [main] INFO
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer: Can't find SST '032501'
> 2023-05-03 18:59:48,891 [main] INFO
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer: Can't find SST '032506'
> 2023-05-03 18:59:48,892 [main] INFO
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer: Can't find SST '032517'
> 2023-05-03 18:59:48,892 [main] INFO
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer: Can't find SST '032525'
> 2023-05-03 18:59:49,270 [main] ERROR
> org.apache.hadoop.ozone.om.OzoneManagerStarter: OM start failed with exception
> java.io.IOException: Snapshot Chain corruption: previous snapshotID given but
> no associated snapshot found in snapshot chain: SnapshotID
> 9384de9d-3e6e-4f18-b4dd-64e69a58f31e
> at
> org.apache.hadoop.ozone.om.SnapshotChainManager.addSnapshotGlobal(SnapshotChainManager.java:86)
> at
> org.apache.hadoop.ozone.om.SnapshotChainManager.addSnapshot(SnapshotChainManager.java:289)
> at
> org.apache.hadoop.ozone.om.SnapshotChainManager.loadFromSnapshotInfoTable(SnapshotChainManager.java:279)
> at
> org.apache.hadoop.ozone.om.SnapshotChainManager.<init>(SnapshotChainManager.java:63)
> at
> org.apache.hadoop.ozone.om.OmMetadataManagerImpl.start(OmMetadataManagerImpl.java:517)
> at
> org.apache.hadoop.ozone.om.OmMetadataManagerImpl.<init>(OmMetadataManagerImpl.java:321)
> at
> org.apache.hadoop.ozone.om.OzoneManager.instantiateServices(OzoneManager.java:762)
> at org.apache.hadoop.ozone.om.OzoneManager.<init>(OzoneManager.java:642)
> at org.apache.hadoop.ozone.om.OzoneManager.createOm(OzoneManager.java:727)
> at
> org.apache.hadoop.ozone.om.OzoneManagerStarter$OMStarterHelper.start(OzoneManagerStarter.java:189)
> at
> org.apache.hadoop.ozone.om.OzoneManagerStarter.startOm(OzoneManagerStarter.java:86)
> at
> org.apache.hadoop.ozone.om.OzoneManagerStarter.call(OzoneManagerStarter.java:74)
> at org.apache.hadoop.hdds.cli.GenericCli.call(GenericCli.java:38)
> at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
> at picocli.CommandLine.access$1300(CommandLine.java:145)
> at
> picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
> at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
> at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
> at
> picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
> at picocli.CommandLine.execute(CommandLine.java:2078)
> at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:100)
> at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:91)
> at
> org.apache.hadoop.ozone.om.OzoneManagerStarter.main(OzoneManagerStarter.java:58)
> 2023-05-03 18:59:49,273 [shutdown-hook-0] INFO
> org.apache.hadoop.ozone.om.OzoneManagerStarter: SHUTDOWN_MSG: {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
