CRC errors are expected in 12.2.7 if you ran 12.2.6 with bluestore. See https://ceph.com/releases/12-2-7-luminous-released/#upgrading-from-v12-2-6
On Fri, Jul 20, 2018 at 8:30 AM Glen Baars <[email protected]> wrote: > > Hello Ceph Users, > > > > We have upgraded all nodes to 12.2.7 now. We have 90PGs ( ~2000 scrub errors > ) to fix from the time when we ran 12.2.6. It doesn’t seem to be affecting > production at this time. > > > > Below is the log of a PG repair. What is the best way to correct these > errors? Is there any further information required? > > > > rados list-inconsistent-obj 1.275 --format=json-pretty > > { > > "epoch": 38481, > > "inconsistents": [] > > } > > > > Is it odd that it doesn’t list any inconsistents? > > > > Ceph.log entries for this PG. > > 2018-07-20 12:13:28.381903 osd.124 osd.124 10.4.35.36:6810/1865422 81 : > cluster [ERR] 1.275 shard 100: soid > 1:ae423e16:::rbd_data.37c2374b0dc51.000000000004917b:head data_digest > 0x1a131dab != data_digest 0x92f2c4c8 from auth oi > 1:ae423e16:::rbd_data.37c2374b0dc51.000000000004917b:head(37917'314836 > client.1079025.0:24453722 dirty|data_digest|omap_digest s 4194304 uv 314836 > dd 92f2c4c8 od ffffffff alloc_hint [4194304 4194304 0]) > > 2018-07-20 12:13:28.381907 osd.124 osd.124 10.4.35.36:6810/1865422 82 : > cluster [ERR] 1.275 shard 124: soid > 1:ae423e16:::rbd_data.37c2374b0dc51.000000000004917b:head data_digest > 0x1a131dab != data_digest 0x92f2c4c8 from auth oi > 1:ae423e16:::rbd_data.37c2374b0dc51.000000000004917b:head(37917'314836 > client.1079025.0:24453722 dirty|data_digest|omap_digest s 4194304 uv 314836 > dd 92f2c4c8 od ffffffff alloc_hint [4194304 4194304 0]) > > 2018-07-20 12:13:28.381909 osd.124 osd.124 10.4.35.36:6810/1865422 83 : > cluster [ERR] 1.275 soid > 1:ae423e16:::rbd_data.37c2374b0dc51.000000000004917b:head: failed to pick > suitable auth object > > 2018-07-20 12:15:15.310579 osd.124 osd.124 10.4.35.36:6810/1865422 84 : > cluster [ERR] 1.275 shard 100: soid > 1:ae455519:::rbd_data.3844874b0dc51.00000000000293f2:head data_digest > 0xdf907335 != data_digest 0x38400b00 from auth oi > 1:ae455519:::rbd_data.3844874b0dc51.00000000000293f2:head(38269'330651 > client.232404.0:23912666 dirty|data_digest|omap_digest s 4194304 uv 307138 dd > 38400b00 od ffffffff alloc_hint [4194304 4194304 0]) > > 2018-07-20 12:15:15.310582 osd.124 osd.124 10.4.35.36:6810/1865422 85 : > cluster [ERR] 1.275 shard 124: soid > 1:ae455519:::rbd_data.3844874b0dc51.00000000000293f2:head data_digest > 0xdf907335 != data_digest 0x38400b00 from auth oi > 1:ae455519:::rbd_data.3844874b0dc51.00000000000293f2:head(38269'330651 > client.232404.0:23912666 dirty|data_digest|omap_digest s 4194304 uv 307138 dd > 38400b00 od ffffffff alloc_hint [4194304 4194304 0]) > > 2018-07-20 12:15:15.310584 osd.124 osd.124 10.4.35.36:6810/1865422 86 : > cluster [ERR] 1.275 soid > 1:ae455519:::rbd_data.3844874b0dc51.00000000000293f2:head: failed to pick > suitable auth object > > 2018-07-20 12:16:07.518970 osd.124 osd.124 10.4.35.36:6810/1865422 87 : > cluster [ERR] 1.275 shard 100: soid > 1:ae470eb2:::rbd_data.37c2374b0dc51.0000000000049a4b:head data_digest > 0x6555a7c9 != data_digest 0xbad822f from auth oi > 1:ae470eb2:::rbd_data.37c2374b0dc51.0000000000049a4b:head(37917'314879 > client.1079025.0:24564045 dirty|data_digest|omap_digest s 4194304 uv 314879 > dd bad822f od ffffffff alloc_hint [4194304 4194304 0]) > > 2018-07-20 12:16:07.518975 osd.124 osd.124 10.4.35.36:6810/1865422 88 : > cluster [ERR] 1.275 shard 124: soid > 1:ae470eb2:::rbd_data.37c2374b0dc51.0000000000049a4b:head data_digest > 0x6555a7c9 != data_digest 0xbad822f from auth oi > 1:ae470eb2:::rbd_data.37c2374b0dc51.0000000000049a4b:head(37917'314879 > client.1079025.0:24564045 dirty|data_digest|omap_digest s 4194304 uv 314879 > dd bad822f od ffffffff alloc_hint [4194304 4194304 0]) > > 2018-07-20 12:16:07.518977 osd.124 osd.124 10.4.35.36:6810/1865422 89 : > cluster [ERR] 1.275 soid > 1:ae470eb2:::rbd_data.37c2374b0dc51.0000000000049a4b:head: failed to pick > suitable auth object > > 2018-07-20 12:16:29.476778 osd.124 osd.124 10.4.35.36:6810/1865422 90 : > cluster [ERR] 1.275 shard 100: soid > 1:ae47e410:::rbd_data.37c2374b0dc51.0000000000024b09:head data_digest > 0xa394e845 != data_digest 0xd8aa931c from auth oi > 1:ae47e410:::rbd_data.37c2374b0dc51.0000000000024b09:head(33683'302224 > client.1079025.0:22963765 dirty|data_digest|omap_digest s 4194304 uv 302224 > dd d8aa931c od ffffffff alloc_hint [4194304 4194304 0]) > > 2018-07-20 12:16:29.476783 osd.124 osd.124 10.4.35.36:6810/1865422 91 : > cluster [ERR] 1.275 shard 124: soid > 1:ae47e410:::rbd_data.37c2374b0dc51.0000000000024b09:head data_digest > 0xa394e845 != data_digest 0xd8aa931c from auth oi > 1:ae47e410:::rbd_data.37c2374b0dc51.0000000000024b09:head(33683'302224 > client.1079025.0:22963765 dirty|data_digest|omap_digest s 4194304 uv 302224 > dd d8aa931c od ffffffff alloc_hint [4194304 4194304 0]) > > 2018-07-20 12:16:29.476787 osd.124 osd.124 10.4.35.36:6810/1865422 92 : > cluster [ERR] 1.275 soid > 1:ae47e410:::rbd_data.37c2374b0dc51.0000000000024b09:head: failed to pick > suitable auth object > > 2018-07-20 12:19:59.498922 osd.124 osd.124 10.4.35.36:6810/1865422 93 : > cluster [ERR] 1.275 shard 100: soid > 1:ae4de127:::rbd_data.37c2374b0dc51.000000000002f6a6:head data_digest > 0x2008cb1b != data_digest 0x218b7cb4 from auth oi > 1:ae4de127:::rbd_data.37c2374b0dc51.000000000002f6a6:head(37426'306744 > client.1079025.0:23363742 dirty|data_digest|omap_digest s 4194304 uv 306744 > dd 218b7cb4 od ffffffff alloc_hint [4194304 4194304 0]) > > 2018-07-20 12:19:59.498925 osd.124 osd.124 10.4.35.36:6810/1865422 94 : > cluster [ERR] 1.275 shard 124: soid > 1:ae4de127:::rbd_data.37c2374b0dc51.000000000002f6a6:head data_digest > 0x2008cb1b != data_digest 0x218b7cb4 from auth oi > 1:ae4de127:::rbd_data.37c2374b0dc51.000000000002f6a6:head(37426'306744 > client.1079025.0:23363742 dirty|data_digest|omap_digest s 4194304 uv 306744 > dd 218b7cb4 od ffffffff alloc_hint [4194304 4194304 0]) > > 2018-07-20 12:19:59.498927 osd.124 osd.124 10.4.35.36:6810/1865422 95 : > cluster [ERR] 1.275 soid > 1:ae4de127:::rbd_data.37c2374b0dc51.000000000002f6a6:head: failed to pick > suitable auth object > > 2018-07-20 12:20:29.937564 osd.124 osd.124 10.4.35.36:6810/1865422 96 : > cluster [ERR] 1.275 shard 100: soid > 1:ae4f1dd8:::rbd_data.7695c59bb0bc2.00000000000005bb:head data_digest > 0x1b42858b != data_digest 0x69a5f3de from auth oi > 1:ae4f1dd8:::rbd_data.7695c59bb0bc2.00000000000005bb:head(38220'328463 > client.1084539.0:403248048 dirty|data_digest|omap_digest s 4194304 uv 308146 > dd 69a5f3de od ffffffff alloc_hint [4194304 4194304 0]) > > 2018-07-20 12:20:29.937568 osd.124 osd.124 10.4.35.36:6810/1865422 97 : > cluster [ERR] 1.275 shard 124: soid > 1:ae4f1dd8:::rbd_data.7695c59bb0bc2.00000000000005bb:head data_digest > 0x1b42858b != data_digest 0x69a5f3de from auth oi > 1:ae4f1dd8:::rbd_data.7695c59bb0bc2.00000000000005bb:head(38220'328463 > client.1084539.0:403248048 dirty|data_digest|omap_digest s 4194304 uv 308146 > dd 69a5f3de od ffffffff alloc_hint [4194304 4194304 0]) > > 2018-07-20 12:20:29.937570 osd.124 osd.124 10.4.35.36:6810/1865422 98 : > cluster [ERR] 1.275 soid > 1:ae4f1dd8:::rbd_data.7695c59bb0bc2.00000000000005bb:head: failed to pick > suitable auth object > > 2018-07-20 12:21:07.463206 osd.124 osd.124 10.4.35.36:6810/1865422 99 : > cluster [ERR] 1.275 repair 12 errors, 0 fixed > > > > Kind regards, > > Glen Baars > > > > From: ceph-users <[email protected]> On Behalf Of Glen Baars > Sent: Wednesday, 18 July 2018 10:33 PM > To: [email protected] > Subject: [ceph-users] 10.2.6 upgrade > > > > Hello Ceph Users, > > > > We installed 12.2.6 on a single node in the cluster ( new node added, 80TB > moved ) > > Disabled scrub/deepscrub once the issues with 12.2.6 were discovered. > > > > Today we upgrade the one affected node to 12.2.7 today, set osd skip data > digest = true and re enabled the scrubs. It’s a 500TB all bluestore cluster. > > > > We are now seeing inconsistent PGs and scrub errors now the scrubbing has > resumed. > > > > What is the best way forward? > > > > Upgrade all nodes to 12.2.7? > Remove the 12.2.7 node and rebuild? > > Kind regards, > > Glen Baars > > BackOnline Manager > > This e-mail is intended solely for the benefit of the addressee(s) and any > other named recipient. It is confidential and may contain legally privileged > or confidential information. If you are not the recipient, any use, > distribution, disclosure or copying of this e-mail is prohibited. The > confidentiality and legal privilege attached to this communication is not > waived or lost by reason of the mistaken transmission or delivery to you. If > you have received this e-mail in error, please notify us immediately. > > This e-mail is intended solely for the benefit of the addressee(s) and any > other named recipient. It is confidential and may contain legally privileged > or confidential information. If you are not the recipient, any use, > distribution, disclosure or copying of this e-mail is prohibited. The > confidentiality and legal privilege attached to this communication is not > waived or lost by reason of the mistaken transmission or delivery to you. If > you have received this e-mail in error, please notify us immediately. > _______________________________________________ > ceph-users mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
