[
https://issues.apache.org/jira/browse/HDDS-10738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841376#comment-17841376
]
Hemant Kumar commented on HDDS-10738:
-------------------------------------
There was no issue till snapshot: e8565f30-1f99-4a2c-a43f-4f253304e7e5.
DoubleBuffer was flushing transitions within a few seconds. The problem starts
from snapshotId e8565f30-1f99-4a2c-a43f-4f253304e7e5 and the time between
snapshot creation to double buffer flush increases from a few seconds to a few
minutes and then a couple of hours.
No flush logs from snapshotId: 875beaeb-745f-41e2-8673-28eb521afbe1 because OM
crashed after that.
|Snapshot creation log time|SnapshotId in snapshot creation log|Double buffer
flush log time|SnapshotId in double buffer flush log|
|2024-04-22 9:51:04,513|54ad92d6-bb44-4c32-8f64-19d9036bca4b|2024-04-22
9:51:09,229|54ad92d6-bb44-4c32-8f64-19d9036bca4b|
|2024-04-22 9:51:04,513|b14d4715-cc79-4083-95e5-27f3d65d5e9e|2024-04-22
9:51:10,312|b14d4715-cc79-4083-95e5-27f3d65d5e9e|
|2024-04-22 9:51:04,513|1819a24d-ecf5-40e9-b512-43e32ba6f9de|2024-04-22
9:51:11,238|1819a24d-ecf5-40e9-b512-43e32ba6f9de|
|2024-04-22 9:51:04,520|9b23e3e4-85ba-408e-9909-7344d89601dd|2024-04-22
9:51:12,348|9b23e3e4-85ba-408e-9909-7344d89601dd|
|2024-04-22 9:51:04,587|58edb1a2-8e65-4776-960f-a6732fd1db46|2024-04-22
9:51:15,192|58edb1a2-8e65-4776-960f-a6732fd1db46|
|2024-04-22 9:51:04,588|5be045c6-4374-4ea9-8576-e98c72a30a04|2024-04-22
9:51:16,584|5be045c6-4374-4ea9-8576-e98c72a30a04|
|2024-04-22 9:51:05,362|279d1556-a8bd-46f4-951d-235f81e57747|2024-04-22
9:51:18,394|279d1556-a8bd-46f4-951d-235f81e57747|
|2024-04-22 9:51:07,379|9e6ff9f1-42b4-4538-9929-8785440dd802|2024-04-22
9:51:19,411|9e6ff9f1-42b4-4538-9929-8785440dd802|
|2024-04-22 9:51:09,29|7ff192e6-b16c-451c-8591-de19a0dca660|2024-04-22
9:51:20,713|7ff192e6-b16c-451c-8591-de19a0dca660|
|2024-04-22 9:51:11,813|97f9b11d-9e7d-44b9-b7e2-f01394c4088c|2024-04-22
9:51:21,983|97f9b11d-9e7d-44b9-b7e2-f01394c4088c|
|2024-04-22 9:51:13,871|5f22a23e-4bbf-45d4-a347-711d0995a4d1|2024-04-22
9:51:23,960|5f22a23e-4bbf-45d4-a347-711d0995a4d1|
|2024-04-22 9:51:14,888|4c494322-2fdf-44b1-a274-214b05f297aa|2024-04-22
9:51:25,2|4c494322-2fdf-44b1-a274-214b05f297aa|
|2024-04-22 9:51:14,963|fce38813-67f1-4754-accd-db6aa0f272e6|2024-04-22
9:51:26,161|fce38813-67f1-4754-accd-db6aa0f272e6|
|2024-04-22 9:51:15,121|9ea96573-1a86-4dbd-9f5d-0535a00ed14b|2024-04-22
9:51:27,162|9ea96573-1a86-4dbd-9f5d-0535a00ed14b|
|2024-04-22 9:51:15,438|9a043bbf-0bf5-4762-9f7d-c9ee80a48832|2024-04-22
9:51:30,748|9a043bbf-0bf5-4762-9f7d-c9ee80a48832|
|2024-04-22 9:51:15,831|6f9b7c67-b8c2-4d86-aaa9-7d3088d2a273|2024-04-22
9:51:31,790|6f9b7c67-b8c2-4d86-aaa9-7d3088d2a273|
|2024-04-22 9:51:16,296|009f7e38-a038-4bc0-a505-bb5948d39898|2024-04-22
9:51:32,924|009f7e38-a038-4bc0-a505-bb5948d39898|
|2024-04-22 9:51:19,830|777ab139-1b3c-4b72-a1b7-12329c3280f4|2024-04-22
9:51:34,474|777ab139-1b3c-4b72-a1b7-12329c3280f4|
|2024-04-22 9:51:22,130|af7edbdb-e4f4-413c-88d8-54af4901627a|2024-04-22
9:51:35,859|af7edbdb-e4f4-413c-88d8-54af4901627a|
|2024-04-22 9:51:23,755|bd2591ef-2d4e-4df6-9395-1a861bcbab6b|2024-04-22
9:51:36,910|bd2591ef-2d4e-4df6-9395-1a861bcbab6b|
|2024-04-22 9:51:28,397|34ca9565-2689-4bad-a0d6-ba162904dacc|2024-04-22
9:51:37,944|34ca9565-2689-4bad-a0d6-ba162904dacc|
|2024-04-22 9:51:30,380|29cfc8bd-43fd-4c63-86b0-603b6667e7fd|2024-04-22
9:51:39,305|29cfc8bd-43fd-4c63-86b0-603b6667e7fd|
|2024-04-22 9:51:30,381|971fbf83-e1c8-4ceb-ae07-93ecc8121b7b|2024-04-22
9:51:40,598|971fbf83-e1c8-4ceb-ae07-93ecc8121b7b|
|2024-04-22 9:51:30,381|8f634462-c6a6-4301-ace4-7152b176f016|2024-04-22
9:51:41,540|8f634462-c6a6-4301-ace4-7152b176f016|
|2024-04-22 9:51:32,563|68ec9ebc-0ad3-4a78-814e-fb943a5af845|2024-04-22
9:51:42,842|68ec9ebc-0ad3-4a78-814e-fb943a5af845|
|2024-04-22 9:51:32,955|f398e781-cdcf-44c7-b31a-b196ce6ab5a0|2024-04-22
9:51:43,893|f398e781-cdcf-44c7-b31a-b196ce6ab5a0|
|2024-04-22 9:51:33,505|f721c709-8b37-4750-a432-7a2c943a1491|2024-04-22
9:51:44,868|f721c709-8b37-4750-a432-7a2c943a1491|
|2024-04-22 9:51:33,780|b144ea05-f90a-400a-bbb8-763d3fda5fbd|2024-04-22
9:51:45,844|b144ea05-f90a-400a-bbb8-763d3fda5fbd|
|2024-04-22 9:51:35,784|7c2ac886-538a-4d9b-88a2-60c9bcce516c|2024-04-22
9:51:46,894|7c2ac886-538a-4d9b-88a2-60c9bcce516c|
|2024-04-22 9:51:36,230|e3598f3c-eb4f-4384-8851-688521a80fdf|2024-04-22
9:51:48,313|e3598f3c-eb4f-4384-8851-688521a80fdf|
|2024-04-22 9:51:36,388|7e30249c-e391-4700-81f3-a30a93353bc1|2024-04-22
9:51:49,748|7e30249c-e391-4700-81f3-a30a93353bc1|
|2024-04-22 9:51:36,530|bd3ecf02-bb15-43a2-b56a-f5ac1811e1ae|2024-04-22
9:51:50,958|bd3ecf02-bb15-43a2-b56a-f5ac1811e1ae|
|2024-04-22 10:43:06,606|b3f34733-ca53-49bf-96bb-9e408419962b|2024-04-22
10:43:08,284|b3f34733-ca53-49bf-96bb-9e408419962b|
|2024-04-22 10:43:06,607|7499926e-a5ef-4e02-92f9-44a8776a7cd9|2024-04-22
10:43:09,243|7499926e-a5ef-4e02-92f9-44a8776a7cd9|
|2024-04-22 10:43:08,440|392b8bd3-5bc4-4f55-aacc-a812935ead31|2024-04-22
10:43:10,427|392b8bd3-5bc4-4f55-aacc-a812935ead31|
|2024-04-22 10:43:08,465|a439bea4-9649-4792-bd45-265a7e5a672a|2024-04-22
10:43:11,378|a439bea4-9649-4792-bd45-265a7e5a672a|
|2024-04-22 10:43:09,720|2613284b-54d9-4695-ab6e-b709b68030cc|2024-04-22
10:43:12,680|2613284b-54d9-4695-ab6e-b709b68030cc|
|2024-04-22 10:43:09,790|d5fd31df-725b-4aa5-ae4b-1030c4bb4c0a|2024-04-22
10:43:13,706|d5fd31df-725b-4aa5-ae4b-1030c4bb4c0a|
|2024-04-22 10:43:09,988|39fda16d-62ae-4034-b457-0fbaf45dd5f7|2024-04-22
10:43:14,848|39fda16d-62ae-4034-b457-0fbaf45dd5f7|
|2024-04-22 10:43:10,320|4444fbfd-cc76-4729-8b79-2a18a07cdd3a|2024-04-22
10:43:15,940|4444fbfd-cc76-4729-8b79-2a18a07cdd3a|
|2024-04-22 10:43:10,637|2824094b-db1b-4912-a3b2-120ef721d47f|2024-04-22
10:43:16,933|2824094b-db1b-4912-a3b2-120ef721d47f|
|2024-04-22 10:43:10,932|e8565f30-1f99-4a2c-a43f-4f253304e7e5|2024-04-22
10:44:19,413|e8565f30-1f99-4a2c-a43f-4f253304e7e5|
|2024-04-22 10:43:11,265|0d04b26b-00ef-4810-a729-b0b19d7097d7|2024-04-22
10:45:20,665|0d04b26b-00ef-4810-a729-b0b19d7097d7|
|2024-04-22 10:43:11,548|3c69933a-da77-4934-8ba4-e6bffee99a2d|2024-04-22
10:46:21,874|3c69933a-da77-4934-8ba4-e6bffee99a2d|
|2024-04-22 10:43:12,780|2c1301f9-30c8-424e-864c-c836867c989b|2024-04-22
10:47:23,351|2c1301f9-30c8-424e-864c-c836867c989b|
|2024-04-22 10:43:13,246|0177d38f-89d2-4b48-abd8-36ed79503ee5|2024-04-22
10:48:24,542|0177d38f-89d2-4b48-abd8-36ed79503ee5|
|2024-04-22 10:43:14,165|0e9241c6-8ec9-4477-9e36-8141868e9392|2024-04-22
10:49:25,820|0e9241c6-8ec9-4477-9e36-8141868e9392|
|2024-04-22 10:43:21,371|3a3c72ce-6ca5-4bcc-91f3-73e9fce20eac|2024-04-22
10:50:27,148|3a3c72ce-6ca5-4bcc-91f3-73e9fce20eac|
|2024-04-22 10:43:26,798|649d8bee-2a2f-498d-bcf9-7cd6c4acdb44|2024-04-22
10:51:28,317|649d8bee-2a2f-498d-bcf9-7cd6c4acdb44|
|2024-04-22 10:43:27,280|f95d0552-aa8e-4d47-bb00-8c5d9a0aade3|2024-04-22
10:52:29,687|f95d0552-aa8e-4d47-bb00-8c5d9a0aade3|
|2024-04-22 10:43:27,281|31813bf8-e836-4d13-9b5b-7e983bf81a76|2024-04-22
10:53:31,23|31813bf8-e836-4d13-9b5b-7e983bf81a76|
|2024-04-22 10:43:27,285|b5346c2a-f432-459a-9598-b70c50cdf14c|2024-04-22
10:54:32,225|b5346c2a-f432-459a-9598-b70c50cdf14c|
|2024-04-22 10:43:27,286|d960a4b4-3d3a-4374-9abb-63868943e74f|2024-04-22
10:55:33,285|d960a4b4-3d3a-4374-9abb-63868943e74f|
|2024-04-22 10:43:27,287|5ed29302-9f88-4af0-97a7-4d4906a1d91a|2024-04-22
10:56:34,546|5ed29302-9f88-4af0-97a7-4d4906a1d91a|
|2024-04-22 10:43:27,292|21473295-9f3d-4fa4-a7cd-4ce5026070cf|2024-04-22
10:57:35,691|21473295-9f3d-4fa4-a7cd-4ce5026070cf|
|2024-04-22 10:43:27,293|710c4754-6d2f-4ad6-8d35-00226d554e8d|2024-04-22
10:58:36,826|710c4754-6d2f-4ad6-8d35-00226d554e8d|
|2024-04-22 10:43:27,293|4e66b330-4916-46d2-b029-1fcc339acf46|2024-04-22
10:59:38,321|4e66b330-4916-46d2-b029-1fcc339acf46|
|2024-04-22 10:43:27,303|6dc129b5-1e6e-4958-ae41-e09cde89b041|2024-04-22
11:00:39,540|6dc129b5-1e6e-4958-ae41-e09cde89b041|
|2024-04-22 10:43:27,304|c4c787ce-cb1e-4e25-bcd2-2a008d5f0e5e|2024-04-22
11:01:40,758|c4c787ce-cb1e-4e25-bcd2-2a008d5f0e5e|
|2024-04-22 10:43:27,305|2fe1333b-d3b3-4703-a5eb-4e813c11894e|2024-04-22
11:02:42,53|2fe1333b-d3b3-4703-a5eb-4e813c11894e|
|2024-04-22 10:43:27,305|fdd544dd-b293-49a0-9240-2d8d3ddf1eb7|2024-04-22
11:03:43,255|fdd544dd-b293-49a0-9240-2d8d3ddf1eb7|
|2024-04-22 10:43:27,305|9eed7234-91e2-43a2-87ca-356b4807c122|2024-04-22
11:04:44,483|9eed7234-91e2-43a2-87ca-356b4807c122|
|2024-04-22 10:43:27,305|ab0cb329-acb8-464c-8add-6921644c11b2|2024-04-22
11:05:45,860|ab0cb329-acb8-464c-8add-6921644c11b2|
|2024-04-22 10:43:28,454|8ddcf4e6-818d-428e-be74-ab3874c0c650|2024-04-22
11:06:47,179|8ddcf4e6-818d-428e-be74-ab3874c0c650|
|2024-04-22 10:43:43,339|3cfb3e62-2414-4bb1-b553-9e43aa6bcef4|2024-04-22
11:07:48,966|3cfb3e62-2414-4bb1-b553-9e43aa6bcef4|
|2024-04-22 10:43:43,606|ba496726-d697-4090-8286-0199d45e0079|2024-04-22
11:08:50,611|ba496726-d697-4090-8286-0199d45e0079|
|2024-04-22 10:43:43,841|9c5e0dbd-f7ce-4651-a447-d925bf4c0dfd|2024-04-22
11:09:52,113|9c5e0dbd-f7ce-4651-a447-d925bf4c0dfd|
|2024-04-22 10:43:44,774|cd7ca9df-b0aa-4c8c-8273-a6ddcc819d9b|2024-04-22
11:10:53,732|cd7ca9df-b0aa-4c8c-8273-a6ddcc819d9b|
|2024-04-22 10:43:45,138|7df63529-561c-44bf-9947-f9d4620e517e|2024-04-22
11:11:54,918|7df63529-561c-44bf-9947-f9d4620e517e|
|2024-04-22 10:43:45,999|a3665acf-1a5c-4841-9ee7-361e8d629ae6|2024-04-22
11:12:56,103|a3665acf-1a5c-4841-9ee7-361e8d629ae6|
|2024-04-22 10:44:05,814|444da8f2-95bd-471e-adfc-354a8aeabc1e|2024-04-22
11:13:57,740|444da8f2-95bd-471e-adfc-354a8aeabc1e|
|2024-04-22 10:44:06,442|113d9924-79b6-408e-b99d-20d4ba918159|2024-04-22
11:14:58,825|113d9924-79b6-408e-b99d-20d4ba918159|
|2024-04-22 10:44:07,564|d95a19f9-91a1-427b-9690-f0fb90d51404|2024-04-22
11:16:00,345|d95a19f9-91a1-427b-9690-f0fb90d51404|
|2024-04-22 10:44:07,709|7e35948e-99a9-4e2b-9018-6adbcd26a4c5|2024-04-22
11:17:01,847|7e35948e-99a9-4e2b-9018-6adbcd26a4c5|
|2024-04-22 10:44:08,297|554816f4-fbb1-4852-9e32-fd44aedf8ffa|2024-04-22
11:18:03,199|554816f4-fbb1-4852-9e32-fd44aedf8ffa|
|2024-04-22 10:44:08,481|f5d150f4-2366-4f0b-9d77-5e6129a540af|2024-04-22
11:19:04,319|f5d150f4-2366-4f0b-9d77-5e6129a540af|
|2024-04-22 10:44:09,48|e1dfa58d-91a3-4134-b26a-522d755a980a|2024-04-22
11:20:05,731|e1dfa58d-91a3-4134-b26a-522d755a980a|
|2024-04-22 10:44:09,331|9b789614-5731-466c-98e1-da5dbb13ba91|2024-04-22
11:21:07,375|9b789614-5731-466c-98e1-da5dbb13ba91|
|2024-04-22 10:44:09,372|15995c0e-36e2-4c5f-80c9-dce0d601fca0|2024-04-22
11:22:08,619|15995c0e-36e2-4c5f-80c9-dce0d601fca0|
|2024-04-22 10:44:09,373|2541fea5-ec37-48f5-a725-75ddff86358b|2024-04-22
11:23:09,647|2541fea5-ec37-48f5-a725-75ddff86358b|
|2024-04-22 10:44:16,365|e3522585-2521-42c9-840d-4064f64e55fd|2024-04-22
11:24:11,509|e3522585-2521-42c9-840d-4064f64e55fd|
|2024-04-22 10:44:16,518|b0f8ac15-44c6-4d75-899b-fb09a3f7309c|2024-04-22
11:25:13,112|b0f8ac15-44c6-4d75-899b-fb09a3f7309c|
|2024-04-22 10:44:16,706|6bc863ea-dcee-4696-acdd-b4b3bf2cca82|2024-04-22
11:26:14,388|6bc863ea-dcee-4696-acdd-b4b3bf2cca82|
|2024-04-22 10:44:17,139|84778ca5-1764-4038-bd9e-26d9888b59c9|2024-04-22
11:27:15,600|84778ca5-1764-4038-bd9e-26d9888b59c9|
|2024-04-22 10:44:17,973|888471e6-5546-40f7-999f-9e78dc8b83e1|2024-04-22
11:28:17,145|888471e6-5546-40f7-999f-9e78dc8b83e1|
|2024-04-22 10:44:18,475|677ab248-6bfe-4f52-9bba-b1c2be372add|2024-04-22
11:29:18,798|677ab248-6bfe-4f52-9bba-b1c2be372add|
|2024-04-22 10:44:28,274|e083b5b2-41ca-442b-a12c-afe0e3592fc5|2024-04-22
11:30:20,376|e083b5b2-41ca-442b-a12c-afe0e3592fc5|
|2024-04-22 10:44:28,392|ef573dc4-c943-434c-bbd0-17abed06cc25|2024-04-22
11:31:21,562|ef573dc4-c943-434c-bbd0-17abed06cc25|
|2024-04-22 10:44:28,392|01148032-7833-49a8-ad03-f2af55a8cfa7|2024-04-22
11:32:22,798|01148032-7833-49a8-ad03-f2af55a8cfa7|
|2024-04-22 10:44:28,393|6b0b2281-5022-49af-b7e5-141384ba12ae|2024-04-22
11:33:24,92|6b0b2281-5022-49af-b7e5-141384ba12ae|
|2024-04-22 10:44:28,395|403df17c-fe68-4f6c-84b9-bccc9eff927e|2024-04-22
11:34:25,245|403df17c-fe68-4f6c-84b9-bccc9eff927e|
|2024-04-22 10:44:28,395|666566fe-f441-4d96-b5ac-e99f9fc6fca3|2024-04-22
11:35:26,307|666566fe-f441-4d96-b5ac-e99f9fc6fca3|
|2024-04-22 10:44:28,398|50e916fd-9565-4eac-b20c-5a05ffb65239|2024-04-22
11:36:28,1|50e916fd-9565-4eac-b20c-5a05ffb65239|
|2024-04-22 10:44:28,748|b5417aa2-b26d-478b-abed-a74f4ad5f133|2024-04-22
11:37:29,103|b5417aa2-b26d-478b-abed-a74f4ad5f133|
|2024-04-22 10:44:29,290|1ec5fc8b-3825-4540-8c11-1a85c589bbf3|2024-04-22
11:38:30,255|1ec5fc8b-3825-4540-8c11-1a85c589bbf3|
|2024-04-22 10:44:29,473|808d45c5-89f6-484c-94ab-f82f5345bb20|2024-04-22
11:39:31,275|808d45c5-89f6-484c-94ab-f82f5345bb20|
|2024-04-22 10:44:29,541|7cb88399-adb6-48ee-99bf-5455dabd2a71|2024-04-22
11:40:32,553|7cb88399-adb6-48ee-99bf-5455dabd2a71|
|2024-04-22 10:44:30,15|d567d6ac-1164-4506-a112-2fc7dea4400e|2024-04-22
11:41:33,756|d567d6ac-1164-4506-a112-2fc7dea4400e|
|2024-04-22 10:44:30,409|c83600ea-76c6-412b-a6ac-9f3989577bcb|2024-04-22
11:42:34,975|c83600ea-76c6-412b-a6ac-9f3989577bcb|
|2024-04-22 10:44:30,590|8d0963da-9673-4961-9c18-4ecc9b3cefc6|2024-04-22
11:43:36,111|8d0963da-9673-4961-9c18-4ecc9b3cefc6|
|2024-04-22 10:44:30,689|f52bd0d6-6f8e-4fc9-9917-81b9aaeb6a32|2024-04-22
12:05:28,942|f52bd0d6-6f8e-4fc9-9917-81b9aaeb6a32|
|2024-04-22 10:44:31,440|7da5b704-43db-4df0-bec8-c417dad6c1d5|2024-04-22
12:06:30,478|7da5b704-43db-4df0-bec8-c417dad6c1d5|
|2024-04-22 10:44:36,124|4ce43a73-44ba-4358-825a-4b047e16db0f|2024-04-22
12:07:32,97|4ce43a73-44ba-4358-825a-4b047e16db0f|
|2024-04-22 10:44:42,784|ca1c4b1a-5419-4233-9095-5358849bbec2|2024-04-22
12:08:33,500|ca1c4b1a-5419-4233-9095-5358849bbec2|
|2024-04-22 10:44:44,132|953c4045-7f26-4239-a273-2793303e1cd0|2024-04-22
12:09:35,395|953c4045-7f26-4239-a273-2793303e1cd0|
|2024-04-22 10:44:47,82|40d02378-cd92-4c60-886f-23a5735ccc8c|2024-04-22
12:19:35,602|40d02378-cd92-4c60-886f-23a5735ccc8c|
|2024-04-22 10:44:48,173|f4d59691-7aff-41ba-8f5e-36736ae6f3ba|2024-04-22
12:20:36,879|f4d59691-7aff-41ba-8f5e-36736ae6f3ba|
|2024-04-22 10:44:54,269|17f338d0-e57c-4f69-afbd-61345eabfad3|2024-04-22
12:21:38,455|17f338d0-e57c-4f69-afbd-61345eabfad3|
|2024-04-22 10:45:08,117|f0bdb167-6c09-4341-8ad7-ae752d311463|2024-04-22
12:22:40,392|f0bdb167-6c09-4341-8ad7-ae752d311463|
|2024-04-22 10:45:08,125|80f5a953-d93a-495c-8277-46744873beab|2024-04-22
12:34:40,292|80f5a953-d93a-495c-8277-46744873beab|
|2024-04-22 10:45:08,435|32912cb9-37c2-4740-9d93-bc0788af05b6|2024-04-22
12:36:40,419|32912cb9-37c2-4740-9d93-bc0788af05b6|
|2024-04-22 10:45:09,266|b9b16628-5973-4610-bac7-39c08da743bf|2024-04-22
12:37:41,689|b9b16628-5973-4610-bac7-39c08da743bf|
|2024-04-22 10:45:09,667|3f3a6c8b-3d5d-479d-89e1-5f4b8403c04e|2024-04-22
12:38:43,534|3f3a6c8b-3d5d-479d-89e1-5f4b8403c04e|
|2024-04-22 10:45:10,558|28a8a93f-50dd-495c-902d-dcc85a67c7b4|2024-04-22
12:39:44,794|28a8a93f-50dd-495c-902d-dcc85a67c7b4|
|2024-04-22 10:45:10,917|3d7bd847-2c8a-4e9c-a7f5-465496688fd7|2024-04-22
12:40:45,970|3d7bd847-2c8a-4e9c-a7f5-465496688fd7|
|2024-04-22 10:45:11,783|b327f4aa-59bb-4423-aaac-d8f313086668|2024-04-22
12:41:47,198|b327f4aa-59bb-4423-aaac-d8f313086668|
|2024-04-22 10:45:11,958|47ef5f8b-f08f-458d-af0f-fc6bd47bd8b5|2024-04-22
12:42:48,749|47ef5f8b-f08f-458d-af0f-fc6bd47bd8b5|
|2024-04-22 10:45:12,737|1db055a7-8831-4f70-a76a-a32b34bbd755|2024-04-22
12:43:49,843|1db055a7-8831-4f70-a76a-a32b34bbd755|
|2024-04-22 10:45:16,278|a136f874-09b3-43d7-90fa-0c14dedb5113|2024-04-22
12:44:51,195|a136f874-09b3-43d7-90fa-0c14dedb5113|
|2024-04-22 10:45:16,975|56519a92-deda-4a78-ba28-9b9d02a6004b|2024-04-22
12:46:51,300|56519a92-deda-4a78-ba28-9b9d02a6004b|
|2024-04-22 10:45:17,670|6c4169a9-fb83-44b2-ad56-864ae23168f4|2024-04-22
12:47:52,902|6c4169a9-fb83-44b2-ad56-864ae23168f4|
|2024-04-22 10:45:18,620|747be8e8-b047-404a-af9b-b3c4b6324462|2024-04-22
12:49:52,923|747be8e8-b047-404a-af9b-b3c4b6324462|
|2024-04-22 10:45:20,75|041c3162-d315-4e4f-ba4c-c6c7662a3399|2024-04-22
12:50:54,242|041c3162-d315-4e4f-ba4c-c6c7662a3399|
|2024-04-22 10:45:21,876|a6861c44-3e71-4902-8d35-a8cb58638850|2024-04-22
12:51:55,510|a6861c44-3e71-4902-8d35-a8cb58638850|
|2024-04-22 10:45:22,217|c2ef9868-6446-4c33-b034-d067b2b4ee63|2024-04-22
12:52:57,514|c2ef9868-6446-4c33-b034-d067b2b4ee63|
|2024-04-22 10:45:26,370|04395b21-4a83-428a-916f-39ff1aca922c|2024-04-22
12:54:57,34|04395b21-4a83-428a-916f-39ff1aca922c|
|2024-04-22 10:45:27,243|f928e9d4-6aa8-4ae0-8a51-64d15bba3767|2024-04-22
12:55:58,569|f928e9d4-6aa8-4ae0-8a51-64d15bba3767|
|2024-04-22 10:45:28,459|d9cd8de1-40e0-4ff1-bd4e-fd3bd724fe97|2024-04-22
12:56:59,714|d9cd8de1-40e0-4ff1-bd4e-fd3bd724fe97|
|2024-04-22 10:45:28,760|9b9208e2-6d4b-459a-9e3a-9cbeccdd0c3a|2024-04-22
12:58:01,100|9b9208e2-6d4b-459a-9e3a-9cbeccdd0c3a|
|2024-04-22 10:45:29,79|c95391c3-4f08-48a8-8cd7-07718cff6afc|2024-04-22
12:59:02,260|c95391c3-4f08-48a8-8cd7-07718cff6afc|
|2024-04-22 10:45:29,142|b30c613d-0548-4535-ba47-7450b27a2ec9|2024-04-22
13:00:03,295|b30c613d-0548-4535-ba47-7450b27a2ec9|
|2024-04-22 10:45:29,220|94bdeb84-7bb6-4840-a417-5bd4e2818d4e|2024-04-22
13:01:04,346|94bdeb84-7bb6-4840-a417-5bd4e2818d4e|
|2024-04-22 10:45:29,280|51d345d4-06f1-439c-a9ff-b6439093f757|2024-04-22
13:02:05,366|51d345d4-06f1-439c-a9ff-b6439093f757|
|2024-04-22 10:45:29,552|dc882064-7bb8-4809-a22d-d83a7c129a12|2024-04-22
13:03:06,552|dc882064-7bb8-4809-a22d-d83a7c129a12|
|2024-04-22 10:45:29,559|777efb72-19f9-4a26-9a64-287ff2a73cf7|2024-04-22
13:04:07,662|777efb72-19f9-4a26-9a64-287ff2a73cf7|
|2024-04-22 10:45:29,592|0975d08e-cb0f-428d-a6b7-ed7e22fd5845|2024-04-22
13:05:08,838|0975d08e-cb0f-428d-a6b7-ed7e22fd5845|
|2024-04-22 10:45:30,184|039da5a8-2ac7-4c80-8858-31441d0e29b6|2024-04-22
13:06:09,759|039da5a8-2ac7-4c80-8858-31441d0e29b6|
|2024-04-22 10:45:32,176|278ac149-0c16-43fa-9518-1a1a65d13f8c|2024-04-22
13:07:11,11|278ac149-0c16-43fa-9518-1a1a65d13f8c|
|2024-04-22 10:45:32,975|536377a2-9400-4606-b6bc-5723e93c2a84|2024-04-22
13:08:12,380|536377a2-9400-4606-b6bc-5723e93c2a84|
|2024-04-22 10:45:33,276|97c1e436-cc27-4512-a157-28e93219889c|2024-04-22
13:10:12,459|97c1e436-cc27-4512-a157-28e93219889c|
|2024-04-22 10:45:42,447|d22c5a57-f509-41cc-8b97-0317d86153a7|2024-04-22
13:11:13,571|d22c5a57-f509-41cc-8b97-0317d86153a7|
|2024-04-22 10:45:42,449|f9b9033b-81ef-4e35-ae35-b347bd55e505|2024-04-22
13:12:15,40|f9b9033b-81ef-4e35-ae35-b347bd55e505|
|2024-04-22 10:45:46,85|305aa7ce-0c5f-4ed8-98f5-d9c84dc79c01|2024-04-22
13:13:16,626|305aa7ce-0c5f-4ed8-98f5-d9c84dc79c01|
|2024-04-22 10:45:46,543|cce1ea43-c703-46c9-a3f3-edf882fcf38e|2024-04-22
13:14:18,2|cce1ea43-c703-46c9-a3f3-edf882fcf38e|
|2024-04-22 10:45:48,214|60b6d6d2-186f-472a-a76d-797aa0bc6cf7|2024-04-22
13:15:19,206|60b6d6d2-186f-472a-a76d-797aa0bc6cf7|
|2024-04-22 10:45:53,211|a1d3c828-baa9-4421-aa96-e68cf6b2c224|2024-04-22
13:16:20,533|a1d3c828-baa9-4421-aa96-e68cf6b2c224|
|2024-04-22 10:46:06,377|875beaeb-745f-41e2-8673-28eb521afbe1|
|2024-04-22 10:46:08,14|1b941cc4-412e-4073-8bba-eeb60a6842ba|
|2024-04-22 10:46:08,673|d84e62d6-85c4-48a1-9c25-0ac3d3f4c6cb|
|2024-04-22 10:46:09,544|090fbc0b-898e-4f8e-bdea-e20fa6eb0efb|
|2024-04-22 10:46:09,936|6f709f43-d203-4875-b721-e04c3253d4b9|
|2024-04-22 10:46:16,295|43bf5019-6003-404a-8311-d1360dabd1fc|
|2024-04-22 10:46:16,461|fcc73ad2-8afa-4aa7-b2ac-4cb7ad91c87d|
|2024-04-22 10:46:17,319|09ce3b3e-1f89-477e-b2d7-022dde127cdc|
|2024-04-22 10:46:17,819|dda8ce06-0582-4218-b301-a6068d7b261d|
|2024-04-22 10:46:18,344|ec3b1a86-1f06-45fe-9361-dce15f543880|
|2024-04-22 10:46:19,836|4e85cb0e-1a3e-47e2-b3de-9b198b7bfc25|
|2024-04-22 10:46:20,139|b053c678-11fa-478f-a22f-d7f6451021af|
> Unable to load snapshot exception encountered in a LR setup
> -----------------------------------------------------------
>
> Key: HDDS-10738
> URL: https://issues.apache.org/jira/browse/HDDS-10738
> Project: Apache Ozone
> Issue Type: Bug
> Components: OM, Snapshot
> Reporter: Jyotirmoy Sinha
> Assignee: Hemant Kumar
> Priority: Major
> Labels: ozone-snapshot
>
> Scenario :
> * Generate data over parallel threads over various volume/buckets
> * Perform parallel snapshot create/delete/list operations over above buckets
> * Perform parallel snapdiff operations over each bucket
> * Perform parallel read operations of snapshot contents
> * Introduce OM and cluster restarts in between along with DN decommissioning
> and balancer restarts.
> Observation - When multiple threads are running with snapshot operations the
> snapshot path contents are not accessible even after 20 mins
> Snapshot creation log OM -
> {code:java}
> 2024-04-22 11:18:49,123 INFO [OM StateMachine ApplyTransaction Thread -
> 0]-org.apache.hadoop.ozone.om.request.snapshot.OMSnapshotCreateRequest:
> Created snapshot: 'snap1713809817' with snapshotId:
> '849a95ab-c5bc-4b78-9d0a-fdad34fd331a' under path 'vol-dp4tz/buck-cp6e6'
> {code}
> OM Error stacktrace -
> {code:java}
> 2024-04-22 11:42:04,768 INFO [IPC Server handler 37 on
> 9862]-org.apache.hadoop.hdds.utils.db.RDBCheckpointUtils: Checkpoint
> directory: 60 didn't get created in
> /var/lib/hadoop-ozone/om/data/db.snapshots/checkpointState/om.db-849a95ab-c5bc-4b78-9d0a-fdad34fd331a
> secs.
> 2024-04-22 11:42:04,768 ERROR [IPC Server handler 37 on
> 9862]-org.apache.hadoop.ozone.om.OmSnapshotManager: Failed to retrieve
> snapshot: /vol-dp4tz/buck-cp6e6/snap1713809817
> TIMEOUT org.apache.hadoop.ozone.om.exceptions.OMException: Unable to load
> snapshot. Snapshot checkpoint directory
> '/var/lib/hadoop-ozone/om/data/db.snapshots/checkpointState/om.db-849a95ab-c5bc-4b78-9d0a-fdad34fd331a'
> does not exist yet. Please wait a few more seconds before retrying
> at
> org.apache.hadoop.ozone.om.snapshot.SnapshotUtils.checkSnapshotDirExist(SnapshotUtils.java:113)
> at
> org.apache.hadoop.ozone.om.OmMetadataManagerImpl.<init>(OmMetadataManagerImpl.java:406)
> at
> org.apache.hadoop.ozone.om.OmSnapshotManager$1.load(OmSnapshotManager.java:357)
> at
> org.apache.hadoop.ozone.om.OmSnapshotManager$1.load(OmSnapshotManager.java:1)
> at
> org.apache.hadoop.ozone.om.snapshot.SnapshotCache.lambda$1(SnapshotCache.java:147)
> at
> java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
> at
> org.apache.hadoop.ozone.om.snapshot.SnapshotCache.get(SnapshotCache.java:143)
> at
> org.apache.hadoop.ozone.om.OmSnapshotManager.checkForSnapshot(OmSnapshotManager.java:625)
> at
> org.apache.hadoop.ozone.om.OzoneManager.getReader(OzoneManager.java:4634)
> at
> org.apache.hadoop.ozone.om.OzoneManager.getFileStatus(OzoneManager.java:3572)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.getOzoneFileStatus(OzoneManagerRequestHandler.java:1002)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleReadRequest(OzoneManagerRequestHandler.java:257)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitReadRequestToOM(OzoneManagerProtocolServerSideTranslatorPB.java:220)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:174)
> at
> org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:143)
> at
> org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:533)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:994)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:922)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2899)
> 2024-04-22 11:42:04,770 WARN [IPC Server handler 37 on
> 9862]-org.apache.hadoop.ipc.Server: IPC Server handler 37 on 9862, call
> Call#2 Retry#0
> org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest from
> 10.17.207.24:57514
> java.lang.IllegalStateException: TIMEOUT
> org.apache.hadoop.ozone.om.exceptions.OMException: Unable to load snapshot.
> Snapshot checkpoint directory
> '/var/lib/hadoop-ozone/om/data/db.snapshots/checkpointState/om.db-849a95ab-c5bc-4b78-9d0a-fdad34fd331a'
> does not exist yet. Please wait a few more seconds before retrying {code}
> The above error is coming for multiple snapshots repeatedly and mostly coming
> in parallel snapshot operations across various volume/buckets, not in serial
> operations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
