Re: [ceph-users] "Failed to authpin" results in large number of blocked requests
http://tracker.ceph.com/issues/25131 may relieve the issue. please try
ceph version 13.2.5.
Regards
Yan, Zheng
On Thu, Mar 28, 2019 at 6:02 PM Zoë O'Connell wrote:
>
> We're running a Ceph mimic (13.2.4) cluster which is predominantly used
> for CephFS. We have recently switched to using multiple active MDSes to
> cope with load on the cluster, but are experiencing problems with large
> numbers of blocked requests when research staff run large experiments.
> The error associated with the block is:
>
> 2019-03-28 09:31:34.246326 [WRN] 6 slow requests, 0 included below;
> oldest blocked for > 423.987868 secs
> 2019-03-28 09:31:29.246202 [WRN] slow request 62.572806 seconds old,
> received at 2019-03-28 09:30:26.673298:
> client_request(client.5882168:1404749 lookup #0x1000441/run_output
> 2019-03-28 09:30:26.653089 caller_uid=0, caller_gid=0{}) currently
> failed to authpin, subtree is being exported
>
> Eventually, many hundreds of requests are blocked for hours.
>
> It appears (As alluded to by the subtree is being exported error) that
> this is related to the MDSes remapping entries between ranks under load,
> as it is always accompanied by messages along the lines of
> "mds.0.migrator nicely exporting to mds.1". Migrations that occur when
> the cluster is not under heavy load complete OK, but under load it seems
> the operation is not completed or entering deadlock for some reason.
>
> We can clear the immediate problem by restarting the affected MDS, and
> have a partial solution by using subtree pinning on everything but this
> is far from ideal. Does anyone have any pointers where else we should
> be looking to troubleshoot this?
>
> Thanks,
>
> Zoe.
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Disable cephx with centralized configs
To debug another issue I'm having, I'd like to try to disable cephx auth between my ceph servers. According to the docs, the way to do this is by setting 'auth_client_required' to 'none' in ceph.conf This no longer works, as you get an error message like this when running 'ceph -s': 2019-04-03 19:29:27.134 7f01772a3700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] If you try to change the setting using 'ceph config', you get both the new and old setting listed in the configuration, and the same error as above. Is it possible to turn off cephx once it it turned on now? If so, what's the right method? -- Shawn Edwards Beware programmers with screwdrivers. They tend to spill them on their keyboards. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] inline_data (was: CephFS and many small files)
On Tue, Apr 2, 2019 at 5:24 AM Clausen, Jörn wrote: > > Hi! > > Am 29.03.2019 um 23:56 schrieb Paul Emmerich: > > There's also some metadata overhead etc. You might want to consider > > enabling inline data in cephfs to handle small files in a > > store-efficient way (note that this feature is officially marked as > > experimental, though). > > http://docs.ceph.com/docs/master/cephfs/experimental-features/#inline-data > > Is there something missing from the documentation? I have turned on this > feature: > > $ ceph fs dump | grep inline_data > dumped fsmap epoch 1224 > inline_data enabled > > I have reduced the size of the bonnie-generated files to 1 byte. But > this is the situation halfway into the test: (output slightly shortened) > > $ rados df > POOL_NAME USED OBJECTS CLONES COPIES > fs-data 3.2 MiB 3390041 0 10170123 > fs-metadata 772 MiB2249 0 6747 > > total_objects3392290 > total_used 643 GiB > total_avail 957 GiB > total_space 1.6 TiB > > i.e. bonnie has created a little over 3 million files, for which the > same number of objects was created in the data pool. So the raw usage is > again at more than 500 GB. Even for inline files, there is one object created in the data pool to hold backtrace information (an xattr of the object) used for hard links and disaster recovery. -- Patrick Donnelly ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] BADAUTHORIZER in Nautilus
ceph-post-file: 789769c4-e7e4-47d3-8fb7-475ea4cfe14a This should have the information you need. On Wed, Apr 3, 2019 at 5:49 PM Sage Weil wrote: > This OSD also appears on teh accepting end of things, and probably > has newer keys that the OSD connecting (tho it' shard to tell > because teh debug level isn't turned up). > > The goal is to find an osd still throwing BADAUTHORIZER messages and then > turn of debug_auth=20 and debug_monc=20. Specifically I'm looking for the > output from MonClient::_check_auth_rotating(), which should tell us > whether the OSD thinks its rotating keys are up to date, or whether it is > having trouble renewing them, or what. It's weird that some OSDs are > using hours-old keys to try to authenticate. :/ > > sage > > > On Wed, 3 Apr 2019, Shawn Edwards wrote: > > > ceph-post-file: 60e07a0c-ee5b-4174-9f51-fa091d5662dc > > > > On Wed, Apr 3, 2019 at 5:30 PM Shawn Edwards > wrote: > > > > > According to ceph versions, all bits are running 14.2.0 > > > > > > I have restarted all of the OSD at least twice and am still getting the > > > same error. > > > > > > I'll send a log file with confirmed interesting bad behavior shortly > > > > > > On Wed, Apr 3, 2019, 17:17 Sage Weil wrote: > > > > > >> 2019-04-03 15:04:01.986 7ffae5778700 10 --1- v1: > 10.36.9.46:6813/5003637 > > >> >> v1:10.36.9.28:6809/8224 conn(0xf6a6000 0x30a02000 :6813 > > >> s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=0).handle_connect_message_2 > > >> authorizor_protocol 2 len 174 > > >> 2019-04-03 15:04:01.986 7ffae5778700 20 AuthRegistry(0xcd64a40) > > >> get_handler peer_type 4 method 2 cluster_methods [2] service_methods > [2] > > >> client_methods [2] > > >> 2019-04-03 15:04:01.986 7ffae5778700 10 cephx: verify_authorizer > > >> decrypted service osd secret_id=41686 > > >> 2019-04-03 15:04:01.986 7ffae5778700 0 auth: could not find > > >> secret_id=41686 > > >> 2019-04-03 15:04:01.986 7ffae5778700 10 auth: dump_rotating: > > >> 2019-04-03 15:04:01.986 7ffae5778700 10 auth: id 41691 ... expires > > >> 2019-04-03 14:43:07.042860 > > >> 2019-04-03 15:04:01.986 7ffae5778700 10 auth: id 41692 ... expires > > >> 2019-04-03 15:43:09.895511 > > >> 2019-04-03 15:04:01.986 7ffae5778700 10 auth: id 41693 ... expires > > >> 2019-04-03 16:43:09.895511 > > >> 2019-04-03 15:04:01.986 7ffae5778700 0 cephx: verify_authorizer could > > >> not get service secret for service osd secret_id=41686 > > >> 2019-04-03 15:04:01.986 7ffae5778700 0 --1- v1: > 10.36.9.46:6813/5003637 > > >> >> v1:10.36.9.28:6809/8224 conn(0xf6a6000 0x30a02000 :6813 > > >> s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=0).handle_connect_message_2: > > >> got bad authorizer, auth_reply_len=0 > > >> > > >> For some reason this OSD has much newer rotating keys than the > > >> connecting OSD. But earlier in the day, this osd was the one > > >> getting BADAUTHORIZER, so maybe that shifted. Can you find an OSD > where > > >> you still see BADAUTHORIZER appearing in the log? > > >> > > >> My guess is that if you restart the OSDs, they'll get fresh rotating > keys > > >> and things will be fine. But that doesn't explain why they're not > > >> renewing on their own right now.. that I'm not so sure about. > > >> > > >> Are your mons all running nautilus? Does 'ceph versions' show > everything > > >> has upgraded? > > >> > > >> sage > > >> > > >> > > >> On Wed, 3 Apr 2019, Shawn Edwards wrote: > > >> > > >> > File uploaded: f1a2bfb3-92b4-495c-8706-f99cb228efc7 > > >> > > > >> > On Wed, Apr 3, 2019 at 4:57 PM Sage Weil wrote: > > >> > > > >> > > Hmm, that doesn't help. > > >> > > > > >> > > Can you set > > >> > > > > >> > > ceph config set osd debug_ms 20 > > >> > > ceph config set osd debug_auth 20 > > >> > > ceph config set osd debug_monc 20 > > >> > > > > >> > > for a few minutes and ceph-post-file the osd logs? (Or send a > private > > >> > > email with a link or something.) > > >> > > > > >> > > Thanks! > > >> > > sage > > >> > > > > >> > > > > >> > > On Wed, 3 Apr 2019, Shawn Edwards wrote: > > >> > > > > >> > > > No strange auth config: > > >> > > > > > >> > > > root@tyr-ceph-mon0:~# ceph config dump | grep -E '(auth|cephx)' > > >> > > > globaladvanced auth_client_required cephx > > >> > > > * > > >> > > > globaladvanced auth_cluster_required cephx > > >> > > > * > > >> > > > globaladvanced auth_service_required cephx > > >> > > > * > > >> > > > > > >> > > > All boxes are using 'minimal' ceph.conf files and centralized > > >> config. > > >> > > > > > >> > > > If you need the full config, it's here: > > >> > > > > https://gist.github.com/lesserevil/3b82d37e517f4561ce53c81629717aae > > >> > > > > > >> > > > On Wed, Apr 3, 2019 at 4:07 PM Sage Weil > wrote: > > >> > > > > > >> > > > > On Wed, 3 Apr 2019, Shawn Edwards wrote: > > >> > > > > >
Re: [ceph-users] BADAUTHORIZER in Nautilus
This OSD also appears on teh accepting end of things, and probably has newer keys that the OSD connecting (tho it' shard to tell because teh debug level isn't turned up). The goal is to find an osd still throwing BADAUTHORIZER messages and then turn of debug_auth=20 and debug_monc=20. Specifically I'm looking for the output from MonClient::_check_auth_rotating(), which should tell us whether the OSD thinks its rotating keys are up to date, or whether it is having trouble renewing them, or what. It's weird that some OSDs are using hours-old keys to try to authenticate. :/ sage On Wed, 3 Apr 2019, Shawn Edwards wrote: > ceph-post-file: 60e07a0c-ee5b-4174-9f51-fa091d5662dc > > On Wed, Apr 3, 2019 at 5:30 PM Shawn Edwards wrote: > > > According to ceph versions, all bits are running 14.2.0 > > > > I have restarted all of the OSD at least twice and am still getting the > > same error. > > > > I'll send a log file with confirmed interesting bad behavior shortly > > > > On Wed, Apr 3, 2019, 17:17 Sage Weil wrote: > > > >> 2019-04-03 15:04:01.986 7ffae5778700 10 --1- v1:10.36.9.46:6813/5003637 > >> >> v1:10.36.9.28:6809/8224 conn(0xf6a6000 0x30a02000 :6813 > >> s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 > >> authorizor_protocol 2 len 174 > >> 2019-04-03 15:04:01.986 7ffae5778700 20 AuthRegistry(0xcd64a40) > >> get_handler peer_type 4 method 2 cluster_methods [2] service_methods [2] > >> client_methods [2] > >> 2019-04-03 15:04:01.986 7ffae5778700 10 cephx: verify_authorizer > >> decrypted service osd secret_id=41686 > >> 2019-04-03 15:04:01.986 7ffae5778700 0 auth: could not find > >> secret_id=41686 > >> 2019-04-03 15:04:01.986 7ffae5778700 10 auth: dump_rotating: > >> 2019-04-03 15:04:01.986 7ffae5778700 10 auth: id 41691 ... expires > >> 2019-04-03 14:43:07.042860 > >> 2019-04-03 15:04:01.986 7ffae5778700 10 auth: id 41692 ... expires > >> 2019-04-03 15:43:09.895511 > >> 2019-04-03 15:04:01.986 7ffae5778700 10 auth: id 41693 ... expires > >> 2019-04-03 16:43:09.895511 > >> 2019-04-03 15:04:01.986 7ffae5778700 0 cephx: verify_authorizer could > >> not get service secret for service osd secret_id=41686 > >> 2019-04-03 15:04:01.986 7ffae5778700 0 --1- v1:10.36.9.46:6813/5003637 > >> >> v1:10.36.9.28:6809/8224 conn(0xf6a6000 0x30a02000 :6813 > >> s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2: > >> got bad authorizer, auth_reply_len=0 > >> > >> For some reason this OSD has much newer rotating keys than the > >> connecting OSD. But earlier in the day, this osd was the one > >> getting BADAUTHORIZER, so maybe that shifted. Can you find an OSD where > >> you still see BADAUTHORIZER appearing in the log? > >> > >> My guess is that if you restart the OSDs, they'll get fresh rotating keys > >> and things will be fine. But that doesn't explain why they're not > >> renewing on their own right now.. that I'm not so sure about. > >> > >> Are your mons all running nautilus? Does 'ceph versions' show everything > >> has upgraded? > >> > >> sage > >> > >> > >> On Wed, 3 Apr 2019, Shawn Edwards wrote: > >> > >> > File uploaded: f1a2bfb3-92b4-495c-8706-f99cb228efc7 > >> > > >> > On Wed, Apr 3, 2019 at 4:57 PM Sage Weil wrote: > >> > > >> > > Hmm, that doesn't help. > >> > > > >> > > Can you set > >> > > > >> > > ceph config set osd debug_ms 20 > >> > > ceph config set osd debug_auth 20 > >> > > ceph config set osd debug_monc 20 > >> > > > >> > > for a few minutes and ceph-post-file the osd logs? (Or send a private > >> > > email with a link or something.) > >> > > > >> > > Thanks! > >> > > sage > >> > > > >> > > > >> > > On Wed, 3 Apr 2019, Shawn Edwards wrote: > >> > > > >> > > > No strange auth config: > >> > > > > >> > > > root@tyr-ceph-mon0:~# ceph config dump | grep -E '(auth|cephx)' > >> > > > globaladvanced auth_client_required cephx > >> > > > * > >> > > > globaladvanced auth_cluster_required cephx > >> > > > * > >> > > > globaladvanced auth_service_required cephx > >> > > > * > >> > > > > >> > > > All boxes are using 'minimal' ceph.conf files and centralized > >> config. > >> > > > > >> > > > If you need the full config, it's here: > >> > > > https://gist.github.com/lesserevil/3b82d37e517f4561ce53c81629717aae > >> > > > > >> > > > On Wed, Apr 3, 2019 at 4:07 PM Sage Weil wrote: > >> > > > > >> > > > > On Wed, 3 Apr 2019, Shawn Edwards wrote: > >> > > > > > Recent nautilus upgrade from mimic. No issues on mimic. > >> > > > > > > >> > > > > > Now getting this or similar in all osd logs, there is very > >> little osd > >> > > > > > communicatoin, and most of the PG are either 'down' or > >> 'unknown', > >> > > even > >> > > > > > though I can see the data on the filestores. > >> > > > > > > >> > > > > > 2019-04-03 13:47:55.280
Re: [ceph-users] BADAUTHORIZER in Nautilus
ceph-post-file: 60e07a0c-ee5b-4174-9f51-fa091d5662dc On Wed, Apr 3, 2019 at 5:30 PM Shawn Edwards wrote: > According to ceph versions, all bits are running 14.2.0 > > I have restarted all of the OSD at least twice and am still getting the > same error. > > I'll send a log file with confirmed interesting bad behavior shortly > > On Wed, Apr 3, 2019, 17:17 Sage Weil wrote: > >> 2019-04-03 15:04:01.986 7ffae5778700 10 --1- v1:10.36.9.46:6813/5003637 >> >> v1:10.36.9.28:6809/8224 conn(0xf6a6000 0x30a02000 :6813 >> s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 >> authorizor_protocol 2 len 174 >> 2019-04-03 15:04:01.986 7ffae5778700 20 AuthRegistry(0xcd64a40) >> get_handler peer_type 4 method 2 cluster_methods [2] service_methods [2] >> client_methods [2] >> 2019-04-03 15:04:01.986 7ffae5778700 10 cephx: verify_authorizer >> decrypted service osd secret_id=41686 >> 2019-04-03 15:04:01.986 7ffae5778700 0 auth: could not find >> secret_id=41686 >> 2019-04-03 15:04:01.986 7ffae5778700 10 auth: dump_rotating: >> 2019-04-03 15:04:01.986 7ffae5778700 10 auth: id 41691 ... expires >> 2019-04-03 14:43:07.042860 >> 2019-04-03 15:04:01.986 7ffae5778700 10 auth: id 41692 ... expires >> 2019-04-03 15:43:09.895511 >> 2019-04-03 15:04:01.986 7ffae5778700 10 auth: id 41693 ... expires >> 2019-04-03 16:43:09.895511 >> 2019-04-03 15:04:01.986 7ffae5778700 0 cephx: verify_authorizer could >> not get service secret for service osd secret_id=41686 >> 2019-04-03 15:04:01.986 7ffae5778700 0 --1- v1:10.36.9.46:6813/5003637 >> >> v1:10.36.9.28:6809/8224 conn(0xf6a6000 0x30a02000 :6813 >> s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2: >> got bad authorizer, auth_reply_len=0 >> >> For some reason this OSD has much newer rotating keys than the >> connecting OSD. But earlier in the day, this osd was the one >> getting BADAUTHORIZER, so maybe that shifted. Can you find an OSD where >> you still see BADAUTHORIZER appearing in the log? >> >> My guess is that if you restart the OSDs, they'll get fresh rotating keys >> and things will be fine. But that doesn't explain why they're not >> renewing on their own right now.. that I'm not so sure about. >> >> Are your mons all running nautilus? Does 'ceph versions' show everything >> has upgraded? >> >> sage >> >> >> On Wed, 3 Apr 2019, Shawn Edwards wrote: >> >> > File uploaded: f1a2bfb3-92b4-495c-8706-f99cb228efc7 >> > >> > On Wed, Apr 3, 2019 at 4:57 PM Sage Weil wrote: >> > >> > > Hmm, that doesn't help. >> > > >> > > Can you set >> > > >> > > ceph config set osd debug_ms 20 >> > > ceph config set osd debug_auth 20 >> > > ceph config set osd debug_monc 20 >> > > >> > > for a few minutes and ceph-post-file the osd logs? (Or send a private >> > > email with a link or something.) >> > > >> > > Thanks! >> > > sage >> > > >> > > >> > > On Wed, 3 Apr 2019, Shawn Edwards wrote: >> > > >> > > > No strange auth config: >> > > > >> > > > root@tyr-ceph-mon0:~# ceph config dump | grep -E '(auth|cephx)' >> > > > globaladvanced auth_client_required cephx >> > > > * >> > > > globaladvanced auth_cluster_required cephx >> > > > * >> > > > globaladvanced auth_service_required cephx >> > > > * >> > > > >> > > > All boxes are using 'minimal' ceph.conf files and centralized >> config. >> > > > >> > > > If you need the full config, it's here: >> > > > https://gist.github.com/lesserevil/3b82d37e517f4561ce53c81629717aae >> > > > >> > > > On Wed, Apr 3, 2019 at 4:07 PM Sage Weil wrote: >> > > > >> > > > > On Wed, 3 Apr 2019, Shawn Edwards wrote: >> > > > > > Recent nautilus upgrade from mimic. No issues on mimic. >> > > > > > >> > > > > > Now getting this or similar in all osd logs, there is very >> little osd >> > > > > > communicatoin, and most of the PG are either 'down' or >> 'unknown', >> > > even >> > > > > > though I can see the data on the filestores. >> > > > > > >> > > > > > 2019-04-03 13:47:55.280 7f13346e3700 0 --1- [v2: >> > > > > > 10.36.9.26:6802/3107,v1:10.36.9.26:6803/3107] >> v1: >> > > 10.36.9.37:6821/8825 >> > > > > > conn(0xa7132000 0xa6b28000 :-1 s=CONNECTING_SEND_CONNECT_MSG >> pgs=0 >> > > cs=0 >> > > > > > l=0).handle_connect_reply_2 connect got BADAUTHORIZER >> > > > > > 2019-04-03 13:47:55.296 7f1333ee2700 0 --1- [v2: >> > > > > > 10.36.9.26:6802/3107,v1:10.36.9.26:6803/3107] >> v1: >> > > > > 10.36.9.37:6841/11204 >> > > > > > conn(0xa9826d00 0xa9b78000 :-1 s=CONNECTING_SEND_CONNECT_MSG >> pgs=0 >> > > cs=0 >> > > > > > l=0).handle_connect_reply_2 connect got BADAUTHORIZER >> > > > > > 2019-04-03 13:47:55.340 7f13346e3700 0 --1- [v2: >> > > > > > 10.36.9.26:6802/3107,v1:10.36.9.26:6803/3107] >> v1: >> > > 10.36.9.37:6829/8425 >> > > > > > conn(0xa7997180 0xaeb22800 :-1 s=CONNECTING_SEND_CONNECT_MSG >> pgs=0 >> > > cs=0
Re: [ceph-users] BADAUTHORIZER in Nautilus
According to ceph versions, all bits are running 14.2.0 I have restarted all of the OSD at least twice and am still getting the same error. I'll send a log file with confirmed interesting bad behavior shortly On Wed, Apr 3, 2019, 17:17 Sage Weil wrote: > 2019-04-03 15:04:01.986 7ffae5778700 10 --1- v1:10.36.9.46:6813/5003637 > >> v1:10.36.9.28:6809/8224 conn(0xf6a6000 0x30a02000 :6813 > s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 > authorizor_protocol 2 len 174 > 2019-04-03 15:04:01.986 7ffae5778700 20 AuthRegistry(0xcd64a40) > get_handler peer_type 4 method 2 cluster_methods [2] service_methods [2] > client_methods [2] > 2019-04-03 15:04:01.986 7ffae5778700 10 cephx: verify_authorizer decrypted > service osd secret_id=41686 > 2019-04-03 15:04:01.986 7ffae5778700 0 auth: could not find > secret_id=41686 > 2019-04-03 15:04:01.986 7ffae5778700 10 auth: dump_rotating: > 2019-04-03 15:04:01.986 7ffae5778700 10 auth: id 41691 ... expires > 2019-04-03 14:43:07.042860 > 2019-04-03 15:04:01.986 7ffae5778700 10 auth: id 41692 ... expires > 2019-04-03 15:43:09.895511 > 2019-04-03 15:04:01.986 7ffae5778700 10 auth: id 41693 ... expires > 2019-04-03 16:43:09.895511 > 2019-04-03 15:04:01.986 7ffae5778700 0 cephx: verify_authorizer could not > get service secret for service osd secret_id=41686 > 2019-04-03 15:04:01.986 7ffae5778700 0 --1- v1:10.36.9.46:6813/5003637 > >> v1:10.36.9.28:6809/8224 conn(0xf6a6000 0x30a02000 :6813 > s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2: > got bad authorizer, auth_reply_len=0 > > For some reason this OSD has much newer rotating keys than the > connecting OSD. But earlier in the day, this osd was the one > getting BADAUTHORIZER, so maybe that shifted. Can you find an OSD where > you still see BADAUTHORIZER appearing in the log? > > My guess is that if you restart the OSDs, they'll get fresh rotating keys > and things will be fine. But that doesn't explain why they're not > renewing on their own right now.. that I'm not so sure about. > > Are your mons all running nautilus? Does 'ceph versions' show everything > has upgraded? > > sage > > > On Wed, 3 Apr 2019, Shawn Edwards wrote: > > > File uploaded: f1a2bfb3-92b4-495c-8706-f99cb228efc7 > > > > On Wed, Apr 3, 2019 at 4:57 PM Sage Weil wrote: > > > > > Hmm, that doesn't help. > > > > > > Can you set > > > > > > ceph config set osd debug_ms 20 > > > ceph config set osd debug_auth 20 > > > ceph config set osd debug_monc 20 > > > > > > for a few minutes and ceph-post-file the osd logs? (Or send a private > > > email with a link or something.) > > > > > > Thanks! > > > sage > > > > > > > > > On Wed, 3 Apr 2019, Shawn Edwards wrote: > > > > > > > No strange auth config: > > > > > > > > root@tyr-ceph-mon0:~# ceph config dump | grep -E '(auth|cephx)' > > > > globaladvanced auth_client_required cephx > > > > * > > > > globaladvanced auth_cluster_required cephx > > > > * > > > > globaladvanced auth_service_required cephx > > > > * > > > > > > > > All boxes are using 'minimal' ceph.conf files and centralized config. > > > > > > > > If you need the full config, it's here: > > > > https://gist.github.com/lesserevil/3b82d37e517f4561ce53c81629717aae > > > > > > > > On Wed, Apr 3, 2019 at 4:07 PM Sage Weil wrote: > > > > > > > > > On Wed, 3 Apr 2019, Shawn Edwards wrote: > > > > > > Recent nautilus upgrade from mimic. No issues on mimic. > > > > > > > > > > > > Now getting this or similar in all osd logs, there is very > little osd > > > > > > communicatoin, and most of the PG are either 'down' or 'unknown', > > > even > > > > > > though I can see the data on the filestores. > > > > > > > > > > > > 2019-04-03 13:47:55.280 7f13346e3700 0 --1- [v2: > > > > > > 10.36.9.26:6802/3107,v1:10.36.9.26:6803/3107] >> v1: > > > 10.36.9.37:6821/8825 > > > > > > conn(0xa7132000 0xa6b28000 :-1 s=CONNECTING_SEND_CONNECT_MSG > pgs=0 > > > cs=0 > > > > > > l=0).handle_connect_reply_2 connect got BADAUTHORIZER > > > > > > 2019-04-03 13:47:55.296 7f1333ee2700 0 --1- [v2: > > > > > > 10.36.9.26:6802/3107,v1:10.36.9.26:6803/3107] >> v1: > > > > > 10.36.9.37:6841/11204 > > > > > > conn(0xa9826d00 0xa9b78000 :-1 s=CONNECTING_SEND_CONNECT_MSG > pgs=0 > > > cs=0 > > > > > > l=0).handle_connect_reply_2 connect got BADAUTHORIZER > > > > > > 2019-04-03 13:47:55.340 7f13346e3700 0 --1- [v2: > > > > > > 10.36.9.26:6802/3107,v1:10.36.9.26:6803/3107] >> v1: > > > 10.36.9.37:6829/8425 > > > > > > conn(0xa7997180 0xaeb22800 :-1 s=CONNECTING_SEND_CONNECT_MSG > pgs=0 > > > cs=0 > > > > > > l=0).handle_connect_reply_2 connect got BADAUTHORIZER > > > > > > 2019-04-03 13:47:55.428 7f1334ee4700 0 auth: could not find > > > > > secret_id=41687 > > > > > > 2019-04-03 13:47:55.428 7f1334ee4700 0 cephx:
Re: [ceph-users] BADAUTHORIZER in Nautilus
2019-04-03 15:04:01.986 7ffae5778700 10 --1- v1:10.36.9.46:6813/5003637 >> v1:10.36.9.28:6809/8224 conn(0xf6a6000 0x30a02000 :6813 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 authorizor_protocol 2 len 174 2019-04-03 15:04:01.986 7ffae5778700 20 AuthRegistry(0xcd64a40) get_handler peer_type 4 method 2 cluster_methods [2] service_methods [2] client_methods [2] 2019-04-03 15:04:01.986 7ffae5778700 10 cephx: verify_authorizer decrypted service osd secret_id=41686 2019-04-03 15:04:01.986 7ffae5778700 0 auth: could not find secret_id=41686 2019-04-03 15:04:01.986 7ffae5778700 10 auth: dump_rotating: 2019-04-03 15:04:01.986 7ffae5778700 10 auth: id 41691 ... expires 2019-04-03 14:43:07.042860 2019-04-03 15:04:01.986 7ffae5778700 10 auth: id 41692 ... expires 2019-04-03 15:43:09.895511 2019-04-03 15:04:01.986 7ffae5778700 10 auth: id 41693 ... expires 2019-04-03 16:43:09.895511 2019-04-03 15:04:01.986 7ffae5778700 0 cephx: verify_authorizer could not get service secret for service osd secret_id=41686 2019-04-03 15:04:01.986 7ffae5778700 0 --1- v1:10.36.9.46:6813/5003637 >> v1:10.36.9.28:6809/8224 conn(0xf6a6000 0x30a02000 :6813 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2: got bad authorizer, auth_reply_len=0 For some reason this OSD has much newer rotating keys than the connecting OSD. But earlier in the day, this osd was the one getting BADAUTHORIZER, so maybe that shifted. Can you find an OSD where you still see BADAUTHORIZER appearing in the log? My guess is that if you restart the OSDs, they'll get fresh rotating keys and things will be fine. But that doesn't explain why they're not renewing on their own right now.. that I'm not so sure about. Are your mons all running nautilus? Does 'ceph versions' show everything has upgraded? sage On Wed, 3 Apr 2019, Shawn Edwards wrote: > File uploaded: f1a2bfb3-92b4-495c-8706-f99cb228efc7 > > On Wed, Apr 3, 2019 at 4:57 PM Sage Weil wrote: > > > Hmm, that doesn't help. > > > > Can you set > > > > ceph config set osd debug_ms 20 > > ceph config set osd debug_auth 20 > > ceph config set osd debug_monc 20 > > > > for a few minutes and ceph-post-file the osd logs? (Or send a private > > email with a link or something.) > > > > Thanks! > > sage > > > > > > On Wed, 3 Apr 2019, Shawn Edwards wrote: > > > > > No strange auth config: > > > > > > root@tyr-ceph-mon0:~# ceph config dump | grep -E '(auth|cephx)' > > > globaladvanced auth_client_required cephx > > > * > > > globaladvanced auth_cluster_required cephx > > > * > > > globaladvanced auth_service_required cephx > > > * > > > > > > All boxes are using 'minimal' ceph.conf files and centralized config. > > > > > > If you need the full config, it's here: > > > https://gist.github.com/lesserevil/3b82d37e517f4561ce53c81629717aae > > > > > > On Wed, Apr 3, 2019 at 4:07 PM Sage Weil wrote: > > > > > > > On Wed, 3 Apr 2019, Shawn Edwards wrote: > > > > > Recent nautilus upgrade from mimic. No issues on mimic. > > > > > > > > > > Now getting this or similar in all osd logs, there is very little osd > > > > > communicatoin, and most of the PG are either 'down' or 'unknown', > > even > > > > > though I can see the data on the filestores. > > > > > > > > > > 2019-04-03 13:47:55.280 7f13346e3700 0 --1- [v2: > > > > > 10.36.9.26:6802/3107,v1:10.36.9.26:6803/3107] >> v1: > > 10.36.9.37:6821/8825 > > > > > conn(0xa7132000 0xa6b28000 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 > > cs=0 > > > > > l=0).handle_connect_reply_2 connect got BADAUTHORIZER > > > > > 2019-04-03 13:47:55.296 7f1333ee2700 0 --1- [v2: > > > > > 10.36.9.26:6802/3107,v1:10.36.9.26:6803/3107] >> v1: > > > > 10.36.9.37:6841/11204 > > > > > conn(0xa9826d00 0xa9b78000 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 > > cs=0 > > > > > l=0).handle_connect_reply_2 connect got BADAUTHORIZER > > > > > 2019-04-03 13:47:55.340 7f13346e3700 0 --1- [v2: > > > > > 10.36.9.26:6802/3107,v1:10.36.9.26:6803/3107] >> v1: > > 10.36.9.37:6829/8425 > > > > > conn(0xa7997180 0xaeb22800 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 > > cs=0 > > > > > l=0).handle_connect_reply_2 connect got BADAUTHORIZER > > > > > 2019-04-03 13:47:55.428 7f1334ee4700 0 auth: could not find > > > > secret_id=41687 > > > > > 2019-04-03 13:47:55.428 7f1334ee4700 0 cephx: verify_authorizer > > could > > > > not > > > > > get service secret for service osd secret_id=41687 > > > > > 2019-04-03 13:47:55.428 7f1334ee4700 0 --1- [v2: > > > > > 10.36.9.26:6802/3107,v1:10.36.9.26:6803/3107] >> v1: > > > > 10.36.9.48:6805/49547 > > > > > conn(0xe02f24480 0xe088cb800 :6803 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH > > > > pgs=0 > > > > > cs=0 l=0).handle_connect_message_2: got bad authorizer, > > auth_reply_len=0 > > > > > > > > > > Thoughts? I
Re: [ceph-users] BADAUTHORIZER in Nautilus
File uploaded: f1a2bfb3-92b4-495c-8706-f99cb228efc7 On Wed, Apr 3, 2019 at 4:57 PM Sage Weil wrote: > Hmm, that doesn't help. > > Can you set > > ceph config set osd debug_ms 20 > ceph config set osd debug_auth 20 > ceph config set osd debug_monc 20 > > for a few minutes and ceph-post-file the osd logs? (Or send a private > email with a link or something.) > > Thanks! > sage > > > On Wed, 3 Apr 2019, Shawn Edwards wrote: > > > No strange auth config: > > > > root@tyr-ceph-mon0:~# ceph config dump | grep -E '(auth|cephx)' > > globaladvanced auth_client_required cephx > > * > > globaladvanced auth_cluster_required cephx > > * > > globaladvanced auth_service_required cephx > > * > > > > All boxes are using 'minimal' ceph.conf files and centralized config. > > > > If you need the full config, it's here: > > https://gist.github.com/lesserevil/3b82d37e517f4561ce53c81629717aae > > > > On Wed, Apr 3, 2019 at 4:07 PM Sage Weil wrote: > > > > > On Wed, 3 Apr 2019, Shawn Edwards wrote: > > > > Recent nautilus upgrade from mimic. No issues on mimic. > > > > > > > > Now getting this or similar in all osd logs, there is very little osd > > > > communicatoin, and most of the PG are either 'down' or 'unknown', > even > > > > though I can see the data on the filestores. > > > > > > > > 2019-04-03 13:47:55.280 7f13346e3700 0 --1- [v2: > > > > 10.36.9.26:6802/3107,v1:10.36.9.26:6803/3107] >> v1: > 10.36.9.37:6821/8825 > > > > conn(0xa7132000 0xa6b28000 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 > cs=0 > > > > l=0).handle_connect_reply_2 connect got BADAUTHORIZER > > > > 2019-04-03 13:47:55.296 7f1333ee2700 0 --1- [v2: > > > > 10.36.9.26:6802/3107,v1:10.36.9.26:6803/3107] >> v1: > > > 10.36.9.37:6841/11204 > > > > conn(0xa9826d00 0xa9b78000 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 > cs=0 > > > > l=0).handle_connect_reply_2 connect got BADAUTHORIZER > > > > 2019-04-03 13:47:55.340 7f13346e3700 0 --1- [v2: > > > > 10.36.9.26:6802/3107,v1:10.36.9.26:6803/3107] >> v1: > 10.36.9.37:6829/8425 > > > > conn(0xa7997180 0xaeb22800 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 > cs=0 > > > > l=0).handle_connect_reply_2 connect got BADAUTHORIZER > > > > 2019-04-03 13:47:55.428 7f1334ee4700 0 auth: could not find > > > secret_id=41687 > > > > 2019-04-03 13:47:55.428 7f1334ee4700 0 cephx: verify_authorizer > could > > > not > > > > get service secret for service osd secret_id=41687 > > > > 2019-04-03 13:47:55.428 7f1334ee4700 0 --1- [v2: > > > > 10.36.9.26:6802/3107,v1:10.36.9.26:6803/3107] >> v1: > > > 10.36.9.48:6805/49547 > > > > conn(0xe02f24480 0xe088cb800 :6803 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH > > > pgs=0 > > > > cs=0 l=0).handle_connect_message_2: got bad authorizer, > auth_reply_len=0 > > > > > > > > Thoughts? I have confirmed that all ceph boxes have good time sync. > > > > > > Do you have any non-default auth-related settings in ceph.conf? > > > > > > sage > > > > > > > > > -- > > Shawn Edwards > > Beware programmers with screwdrivers. They tend to spill them on their > > keyboards. > > > -- Shawn Edwards Beware programmers with screwdrivers. They tend to spill them on their keyboards. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] BADAUTHORIZER in Nautilus
Hmm, that doesn't help. Can you set ceph config set osd debug_ms 20 ceph config set osd debug_auth 20 ceph config set osd debug_monc 20 for a few minutes and ceph-post-file the osd logs? (Or send a private email with a link or something.) Thanks! sage On Wed, 3 Apr 2019, Shawn Edwards wrote: > No strange auth config: > > root@tyr-ceph-mon0:~# ceph config dump | grep -E '(auth|cephx)' > globaladvanced auth_client_required cephx > * > globaladvanced auth_cluster_required cephx > * > globaladvanced auth_service_required cephx > * > > All boxes are using 'minimal' ceph.conf files and centralized config. > > If you need the full config, it's here: > https://gist.github.com/lesserevil/3b82d37e517f4561ce53c81629717aae > > On Wed, Apr 3, 2019 at 4:07 PM Sage Weil wrote: > > > On Wed, 3 Apr 2019, Shawn Edwards wrote: > > > Recent nautilus upgrade from mimic. No issues on mimic. > > > > > > Now getting this or similar in all osd logs, there is very little osd > > > communicatoin, and most of the PG are either 'down' or 'unknown', even > > > though I can see the data on the filestores. > > > > > > 2019-04-03 13:47:55.280 7f13346e3700 0 --1- [v2: > > > 10.36.9.26:6802/3107,v1:10.36.9.26:6803/3107] >> v1:10.36.9.37:6821/8825 > > > conn(0xa7132000 0xa6b28000 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 > > > l=0).handle_connect_reply_2 connect got BADAUTHORIZER > > > 2019-04-03 13:47:55.296 7f1333ee2700 0 --1- [v2: > > > 10.36.9.26:6802/3107,v1:10.36.9.26:6803/3107] >> v1: > > 10.36.9.37:6841/11204 > > > conn(0xa9826d00 0xa9b78000 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 > > > l=0).handle_connect_reply_2 connect got BADAUTHORIZER > > > 2019-04-03 13:47:55.340 7f13346e3700 0 --1- [v2: > > > 10.36.9.26:6802/3107,v1:10.36.9.26:6803/3107] >> v1:10.36.9.37:6829/8425 > > > conn(0xa7997180 0xaeb22800 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 > > > l=0).handle_connect_reply_2 connect got BADAUTHORIZER > > > 2019-04-03 13:47:55.428 7f1334ee4700 0 auth: could not find > > secret_id=41687 > > > 2019-04-03 13:47:55.428 7f1334ee4700 0 cephx: verify_authorizer could > > not > > > get service secret for service osd secret_id=41687 > > > 2019-04-03 13:47:55.428 7f1334ee4700 0 --1- [v2: > > > 10.36.9.26:6802/3107,v1:10.36.9.26:6803/3107] >> v1: > > 10.36.9.48:6805/49547 > > > conn(0xe02f24480 0xe088cb800 :6803 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH > > pgs=0 > > > cs=0 l=0).handle_connect_message_2: got bad authorizer, auth_reply_len=0 > > > > > > Thoughts? I have confirmed that all ceph boxes have good time sync. > > > > Do you have any non-default auth-related settings in ceph.conf? > > > > sage > > > > > -- > Shawn Edwards > Beware programmers with screwdrivers. They tend to spill them on their > keyboards. > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] BADAUTHORIZER in Nautilus
No strange auth config: root@tyr-ceph-mon0:~# ceph config dump | grep -E '(auth|cephx)' globaladvanced auth_client_required cephx * globaladvanced auth_cluster_required cephx * globaladvanced auth_service_required cephx * All boxes are using 'minimal' ceph.conf files and centralized config. If you need the full config, it's here: https://gist.github.com/lesserevil/3b82d37e517f4561ce53c81629717aae On Wed, Apr 3, 2019 at 4:07 PM Sage Weil wrote: > On Wed, 3 Apr 2019, Shawn Edwards wrote: > > Recent nautilus upgrade from mimic. No issues on mimic. > > > > Now getting this or similar in all osd logs, there is very little osd > > communicatoin, and most of the PG are either 'down' or 'unknown', even > > though I can see the data on the filestores. > > > > 2019-04-03 13:47:55.280 7f13346e3700 0 --1- [v2: > > 10.36.9.26:6802/3107,v1:10.36.9.26:6803/3107] >> v1:10.36.9.37:6821/8825 > > conn(0xa7132000 0xa6b28000 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 > > l=0).handle_connect_reply_2 connect got BADAUTHORIZER > > 2019-04-03 13:47:55.296 7f1333ee2700 0 --1- [v2: > > 10.36.9.26:6802/3107,v1:10.36.9.26:6803/3107] >> v1: > 10.36.9.37:6841/11204 > > conn(0xa9826d00 0xa9b78000 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 > > l=0).handle_connect_reply_2 connect got BADAUTHORIZER > > 2019-04-03 13:47:55.340 7f13346e3700 0 --1- [v2: > > 10.36.9.26:6802/3107,v1:10.36.9.26:6803/3107] >> v1:10.36.9.37:6829/8425 > > conn(0xa7997180 0xaeb22800 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 > > l=0).handle_connect_reply_2 connect got BADAUTHORIZER > > 2019-04-03 13:47:55.428 7f1334ee4700 0 auth: could not find > secret_id=41687 > > 2019-04-03 13:47:55.428 7f1334ee4700 0 cephx: verify_authorizer could > not > > get service secret for service osd secret_id=41687 > > 2019-04-03 13:47:55.428 7f1334ee4700 0 --1- [v2: > > 10.36.9.26:6802/3107,v1:10.36.9.26:6803/3107] >> v1: > 10.36.9.48:6805/49547 > > conn(0xe02f24480 0xe088cb800 :6803 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH > pgs=0 > > cs=0 l=0).handle_connect_message_2: got bad authorizer, auth_reply_len=0 > > > > Thoughts? I have confirmed that all ceph boxes have good time sync. > > Do you have any non-default auth-related settings in ceph.conf? > > sage > -- Shawn Edwards Beware programmers with screwdrivers. They tend to spill them on their keyboards. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] BADAUTHORIZER in Nautilus
On Wed, 3 Apr 2019, Shawn Edwards wrote: > Recent nautilus upgrade from mimic. No issues on mimic. > > Now getting this or similar in all osd logs, there is very little osd > communicatoin, and most of the PG are either 'down' or 'unknown', even > though I can see the data on the filestores. > > 2019-04-03 13:47:55.280 7f13346e3700 0 --1- [v2: > 10.36.9.26:6802/3107,v1:10.36.9.26:6803/3107] >> v1:10.36.9.37:6821/8825 > conn(0xa7132000 0xa6b28000 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 > l=0).handle_connect_reply_2 connect got BADAUTHORIZER > 2019-04-03 13:47:55.296 7f1333ee2700 0 --1- [v2: > 10.36.9.26:6802/3107,v1:10.36.9.26:6803/3107] >> v1:10.36.9.37:6841/11204 > conn(0xa9826d00 0xa9b78000 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 > l=0).handle_connect_reply_2 connect got BADAUTHORIZER > 2019-04-03 13:47:55.340 7f13346e3700 0 --1- [v2: > 10.36.9.26:6802/3107,v1:10.36.9.26:6803/3107] >> v1:10.36.9.37:6829/8425 > conn(0xa7997180 0xaeb22800 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 > l=0).handle_connect_reply_2 connect got BADAUTHORIZER > 2019-04-03 13:47:55.428 7f1334ee4700 0 auth: could not find secret_id=41687 > 2019-04-03 13:47:55.428 7f1334ee4700 0 cephx: verify_authorizer could not > get service secret for service osd secret_id=41687 > 2019-04-03 13:47:55.428 7f1334ee4700 0 --1- [v2: > 10.36.9.26:6802/3107,v1:10.36.9.26:6803/3107] >> v1:10.36.9.48:6805/49547 > conn(0xe02f24480 0xe088cb800 :6803 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 > cs=0 l=0).handle_connect_message_2: got bad authorizer, auth_reply_len=0 > > Thoughts? I have confirmed that all ceph boxes have good time sync. Do you have any non-default auth-related settings in ceph.conf? sage ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] How to tune Ceph RBD mirroring parameters to speed up replication
For better or worse, out of the box, librbd and rbd-mirror are configured to conserve memory at the expense of performance to support the potential case of thousands of images being mirrored and only a single "rbd-mirror" daemon attempting to handle the load. You can optimize writes by adding "rbd_journal_max_payload_bytes = 8388608" to the "[client]" section on the librbd client nodes. Normally, writes larger than 16KiB are broken into multiple journal entries to allow the remote "rbd-mirror" daemon to make forward progress w/o using too much memory, so this will ensure large IOs only require a single journal entry. You can also add "rbd_mirror_journal_max_fetch_bytes = 33554432" to the "[client]" section on the "rbd-mirror" daemon nodes and restart the daemon for the change to take effect. Normally, the daemon tries to nibble the per-image journal events to prevent excessive memory use in the case where potentially thousands of images are being mirrored. On Wed, Apr 3, 2019 at 4:34 PM huxia...@horebdata.cn wrote: > > Hello, folks, > > I am setting up two ceph clusters to test async replication via RBD > mirroring. The two clusters are very close, just in two buildings about 20m > away, and the networking is very good as well, 10Gb Fiber connection. In this > case, how should i tune the relevant RBD mirroring parameters to accelerate > the replication? > > thanks in advance, > > Samuel > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Jason ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] BADAUTHORIZER in Nautilus
Recent nautilus upgrade from mimic. No issues on mimic. Now getting this or similar in all osd logs, there is very little osd communicatoin, and most of the PG are either 'down' or 'unknown', even though I can see the data on the filestores. 2019-04-03 13:47:55.280 7f13346e3700 0 --1- [v2: 10.36.9.26:6802/3107,v1:10.36.9.26:6803/3107] >> v1:10.36.9.37:6821/8825 conn(0xa7132000 0xa6b28000 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_2 connect got BADAUTHORIZER 2019-04-03 13:47:55.296 7f1333ee2700 0 --1- [v2: 10.36.9.26:6802/3107,v1:10.36.9.26:6803/3107] >> v1:10.36.9.37:6841/11204 conn(0xa9826d00 0xa9b78000 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_2 connect got BADAUTHORIZER 2019-04-03 13:47:55.340 7f13346e3700 0 --1- [v2: 10.36.9.26:6802/3107,v1:10.36.9.26:6803/3107] >> v1:10.36.9.37:6829/8425 conn(0xa7997180 0xaeb22800 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_2 connect got BADAUTHORIZER 2019-04-03 13:47:55.428 7f1334ee4700 0 auth: could not find secret_id=41687 2019-04-03 13:47:55.428 7f1334ee4700 0 cephx: verify_authorizer could not get service secret for service osd secret_id=41687 2019-04-03 13:47:55.428 7f1334ee4700 0 --1- [v2: 10.36.9.26:6802/3107,v1:10.36.9.26:6803/3107] >> v1:10.36.9.48:6805/49547 conn(0xe02f24480 0xe088cb800 :6803 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2: got bad authorizer, auth_reply_len=0 Thoughts? I have confirmed that all ceph boxes have good time sync. -- Shawn Edwards Beware programmers with screwdrivers. They tend to spill them on their keyboards. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] How to tune Ceph RBD mirroring parameters to speed up replication
Hello, folks, I am setting up two ceph clusters to test async replication via RBD mirroring. The two clusters are very close, just in two buildings about 20m away, and the networking is very good as well, 10Gb Fiber connection. In this case, how should i tune the relevant RBD mirroring parameters to accelerate the replication? thanks in advance, Samuel ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] MDS allocates all memory (>500G) replaying, OOM-killed, repeat
`ceph versions` reports:
{
"mon": {
"ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0)
luminous (stable)": 3
},
"mgr": {
"ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0)
luminous (stable)": 3
},
"osd": {
"ceph version 12.2.10 (177915764b752804194937482a39e95e0ca3de94)
luminous (stable)": 197,
"ceph version 12.2.9 (9e300932ef8a8916fb3fda78c58691a6ab0f4217)
luminous (stable)": 11
},
"mds": {
"ceph version 12.2.10 (177915764b752804194937482a39e95e0ca3de94)
luminous (stable)": 2,
"ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0)
luminous (stable)": 1
},
"overall": {
"ceph version 12.2.10 (177915764b752804194937482a39e95e0ca3de94)
luminous (stable)": 199,
"ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0)
luminous (stable)": 7,
"ceph version 12.2.9 (9e300932ef8a8916fb3fda78c58691a6ab0f4217)
luminous (stable)": 11
}
}
I didn't realize we were in such a weird state with versions, we'll update all
those to 12.2.10 today :)
From: Yan, Zheng
Sent: Tuesday, April 2, 2019 20:26
To: Sergey Malinin
Cc: Pickett, Neale T; ceph-users
Subject: Re: [ceph-users] MDS allocates all memory (>500G) replaying,
OOM-killed, repeat
Looks like http://tracker.ceph.com/issues/37399. which version of
ceph-mds do you use?
On Tue, Apr 2, 2019 at 7:47 AM Sergey Malinin wrote:
>
> These steps pretty well correspond to
> http://docs.ceph.com/docs/mimic/cephfs/disaster-recovery/
> Were you able to replay journal manually with no issues? IIRC,
> "cephfs-journal-tool recover_dentries" would lead to OOM in case of MDS doing
> so, and it has already been discussed on this list.
>
>
> April 2, 2019 1:37 AM, "Pickett, Neale T" wrote:
>
> Here is what I wound up doing to fix this:
>
> Bring down all MDSes so they stop flapping
> Back up journal (as seen in previous message)
> Apply journal manually
> Reset journal manually
> Clear session table
> Clear other tables (not sure I needed to do this)
> Mark FS down
> Mark the rank 0 MDS as failed
> Reset the FS (yes, I really mean it)
> Restart MDSes
> Finally get some sleep
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] x pgs not deep-scrubbed in time
Hi Alex, I'm okay myself with the number of scrubs performed, would you expect tweaking any of those values to let the deep-scrubs finish in time/ Thanks, Michael On Wed, 3 Apr 2019 at 10:30, Alexandru Cucu wrote: > Hello, > > You can increase *osd scrub max interval* and *osd deep scrub > interval* if you don't want at least one scrub/deep scrub per week. > > I would also play with *osd max scrubs* and *osd scrub load threshold* > to do more scrubbing work, but be careful as it will have a huge > impact on performance. > > --- > Alex Cucu > > On Wed, Apr 3, 2019 at 3:46 PM Michael Sudnick > wrote: > > > > Hello, was on IRC yesterday about this and got some input, but haven't > figured out a solution yet. I have a 5 node, 41 OSD cluster which currently > has the warning "295 pgs not deep-scrubbed in time". The number slowly > increases as deep scrubs happen. In my cluster I'm primarily using 5400 RPM > 2.5" disks, and that's my general bottleneck. Processors are 8/16 core > Intel® Xeon processor D-1541. 8 OSDs per node (one has 9), and each node > hosts a MON, MGR and MDS. > > > > My CPU usage is low, it's a very low traffic cluster, just a home lab. > CPU usage rarely spikes around 30%. RAM is fine, each node has 64GiB, and > only about 33GiB is used. Network is overkill, 2x1GbE public, and 2x10GbE > cluster. Disk %util when deep scrubs are happening can hit 80%, so that > seems to be my bottleneck. > > > > I am running Nautilus 14.2.0. I've been running fine since release up to > about 3 days ago where I had a disk die and replaced it. > > > > Any suggestions on what I can do? Thank you for any suggestions. > > > > -Michael > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] x pgs not deep-scrubbed in time
Hello, You can increase *osd scrub max interval* and *osd deep scrub interval* if you don't want at least one scrub/deep scrub per week. I would also play with *osd max scrubs* and *osd scrub load threshold* to do more scrubbing work, but be careful as it will have a huge impact on performance. --- Alex Cucu On Wed, Apr 3, 2019 at 3:46 PM Michael Sudnick wrote: > > Hello, was on IRC yesterday about this and got some input, but haven't > figured out a solution yet. I have a 5 node, 41 OSD cluster which currently > has the warning "295 pgs not deep-scrubbed in time". The number slowly > increases as deep scrubs happen. In my cluster I'm primarily using 5400 RPM > 2.5" disks, and that's my general bottleneck. Processors are 8/16 core Intel® > Xeon processor D-1541. 8 OSDs per node (one has 9), and each node hosts a > MON, MGR and MDS. > > My CPU usage is low, it's a very low traffic cluster, just a home lab. CPU > usage rarely spikes around 30%. RAM is fine, each node has 64GiB, and only > about 33GiB is used. Network is overkill, 2x1GbE public, and 2x10GbE cluster. > Disk %util when deep scrubs are happening can hit 80%, so that seems to be my > bottleneck. > > I am running Nautilus 14.2.0. I've been running fine since release up to > about 3 days ago where I had a disk die and replaced it. > > Any suggestions on what I can do? Thank you for any suggestions. > > -Michael > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph-iscsi: (Config.lock) Timed out (30s) waiting for excl lock on gateway.conf object
running "rados -p rbd lock list gateway.conf" gives:
{"objname":"gateway.conf","locks":[{"name":"lock"}]}
To be sure I stopped all related services (tcmu-runner, rbd-target-gw,
rbd-target-api) on both gateways and ran "rados -p rbd lock list
gateway.conf" again, result was the same as above. Honestly I don't know
what to do with that...
To show a more complete picture I'm posting more logs below when trying
to create another disk in gwcli with "cluster_client_name =
client.iscsi" in /etc/ceph/iscsi-gateway.cfg.
The main question is:
does the gateway work with a user other than client.admin (e.g.
client.iscsi)?
if yes: what privileges does this user need?
thx
matthias
Apr 3 15:52:31 ceiscsi0 tcmu-runner: tcmu_notify_lock_lost:222
rbd/iscsi.test01: Async lock drop. Old state 1
Apr 3 15:52:31 ceiscsi0 tcmu-runner: alua_implicit_transition:566
rbd/iscsi.test01: Starting lock acquisition operation.
Apr 3 15:52:31 ceiscsi0 tcmu-runner: tcmu_rbd_lock:762
rbd/iscsi.test01: Acquired exclusive lock.
Apr 3 15:52:44 ceiscsi0 journal: ::1 - - [03/Apr/2019 15:52:44] "GET
/api/config HTTP/1.1" 200 -
Apr 3 15:52:44 ceiscsi0 journal: ::1 - - [03/Apr/2019 15:52:44] "GET
/api/config HTTP/1.1" 200 -
Apr 3 15:52:44 ceiscsi0 journal: :::xxx.xxx.215.61 - - [03/Apr/2019
15:52:44] "GET /api/_ping HTTP/1.1" 200 -
Apr 3 15:52:44 ceiscsi0 journal: :::xxx.xxx.215.61 - - [03/Apr/2019
15:52:44] "GET /api/_ping HTTP/1.1" 200 -
Apr 3 15:52:48 ceiscsi0 tcmu-runner: tcmu_notify_lock_lost:222
rbd/iscsi.test01: Async lock drop. Old state 1
Apr 3 15:52:48 ceiscsi0 tcmu-runner: alua_implicit_transition:566
rbd/iscsi.test01: Starting lock acquisition operation.
Apr 3 15:52:49 ceiscsi0 tcmu-runner: tcmu_rbd_lock:762
rbd/iscsi.test01: Acquired exclusive lock.
Apr 3 15:52:54 ceiscsi0 journal: ::1 - - [03/Apr/2019 15:52:54] "GET
/api/_ping HTTP/1.1" 200 -
Apr 3 15:52:54 ceiscsi0 journal: (LUN.add_dev_to_lio) Adding image
'iscsi/test02' to LIO backstore user:rbd
Apr 3 15:52:54 ceiscsi0 journal: (LUN.add_dev_to_lio) Successfully
added iscsi/test02 to LIO
Apr 3 15:52:54 ceiscsi0 journal: (LUN.allocate) added 'iscsi/test02' to
LIO and config object
Apr 3 15:52:54 ceiscsi0 journal: :::xxx.xxx.215.61 - - [03/Apr/2019
15:52:54] "GET /api/_ping HTTP/1.1" 200 -
Apr 3 15:53:04 ceiscsi0 journal: :::xxx.xxx.215.61 - - [03/Apr/2019
15:53:04] "GET /api/_ping HTTP/1.1" 200 -
Apr 3 15:53:13 ceiscsi0 tcmu-runner: tcmu_notify_lock_lost:222
rbd/iscsi.test01: Async lock drop. Old state 1
Apr 3 15:53:13 ceiscsi0 tcmu-runner: alua_implicit_transition:566
rbd/iscsi.test01: Starting lock acquisition operation.
Apr 3 15:53:13 ceiscsi0 tcmu-runner: tcmu_rbd_lock:762
rbd/iscsi.test01: Acquired exclusive lock.
Apr 3 15:53:14 ceiscsi0 journal: :::xxx.xxx.215.61 - - [03/Apr/2019
15:53:14] "GET /api/_ping HTTP/1.1" 200 -
Apr 3 15:53:24 ceiscsi0 journal: :::xxx.xxx.215.61 - - [03/Apr/2019
15:53:24] "GET /api/_ping HTTP/1.1" 200 -
Apr 3 15:53:24 ceiscsi0 journal: (Config.lock) Timed out (30s) waiting
for excl lock on gateway.conf object
Apr 3 15:53:24 ceiscsi0 journal: LUN alloc problem - Timed out (30s)
waiting for excl lock on gateway.conf object
Apr 3 15:53:24 ceiscsi0 journal: ::1 - - [03/Apr/2019 15:53:24] "PUT
/api/_disk/iscsi/test02 HTTP/1.1" 500 -
Apr 3 15:53:24 ceiscsi0 journal: _disk change on localhost failed with 500
Apr 3 15:53:24 ceiscsi0 journal: ::1 - - [03/Apr/2019 15:53:24] "PUT
/api/disk/iscsi/test02 HTTP/1.1" 500 -
Apr 3 15:53:34 ceiscsi0 tcmu-runner: tcmu_notify_lock_lost:222
rbd/iscsi.test01: Async lock drop. Old state 1
Apr 3 15:53:34 ceiscsi0 tcmu-runner: alua_implicit_transition:566
rbd/iscsi.test01: Starting lock acquisition operation.
Apr 3 15:53:34 ceiscsi0 journal: :::xxx.xxx.215.61 - - [03/Apr/2019
15:53:34] "GET /api/_ping HTTP/1.1" 200 -
Apr 3 15:53:35 ceiscsi0 tcmu-runner: tcmu_rbd_lock:762
rbd/iscsi.test01: Acquired exclusive lock.
Am 01.04.19 um 17:05 schrieb Jason Dillaman:
What happens when you run "rados -p rbd lock list gateway.conf"?
On Fri, Mar 29, 2019 at 12:19 PM Matthias Leopold
wrote:
Hi,
I upgraded my test Ceph iSCSI gateways to
ceph-iscsi-3.0-6.g433bbaa.el7.noarch.
I'm trying to use the new parameter "cluster_client_name", which - to me
- sounds like I don't have to access the ceph cluster as "client.admin"
anymore. I created a "client.iscsi" user and watched what happened. The
gateways can obviously read the config (which I created when I was still
client.admin), but when I try to change anything (like create a new disk
in pool "iscsi") I get the following error:
(Config.lock) Timed out (30s) waiting for excl lock on gateway.conf object
I suspect this is related to the privileges of "client.iscsi", but I
couldn't find the correct settings yet. The last thing I tried was:
caps: [mon] allow r, allow command "osd blacklist"
caps: [osd] allow * pool=rbd, profile rbd pool=iscsi
Can anybody tell me how to solve this?
My Ceph
[ceph-users] Wrong certificate delivered on https://ceph.io/
Dear all, is there somebody in charge of the ceph hosting here, or someone who knows the guy who knows another guy who may know... Saw this while reading the FOSDEM 2019 presentation by Sage, I clicked on the link at the end which is labelled http://ceph.io/ but linked to https://ceph.io/. The certificate delivered when you visit https://ceph.io/ is only valid for ceph.com and www.ceph.com. If someone can fix this or point me to the right person. Best, Raphaël P.S. Thank you so much for Ceph ;) ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] x pgs not deep-scrubbed in time
Hello, was on IRC yesterday about this and got some input, but haven't figured out a solution yet. I have a 5 node, 41 OSD cluster which currently has the warning "295 pgs not deep-scrubbed in time". The number slowly increases as deep scrubs happen. In my cluster I'm primarily using 5400 RPM 2.5" disks, and that's my general bottleneck. Processors are 8/16 core Intel® Xeon processor D-1541. 8 OSDs per node (one has 9), and each node hosts a MON, MGR and MDS. My CPU usage is low, it's a very low traffic cluster, just a home lab. CPU usage rarely spikes around 30%. RAM is fine, each node has 64GiB, and only about 33GiB is used. Network is overkill, 2x1GbE public, and 2x10GbE cluster. Disk %util when deep scrubs are happening can hit 80%, so that seems to be my bottleneck. I am running Nautilus 14.2.0. I've been running fine since release up to about 3 days ago where I had a disk die and replaced it. Any suggestions on what I can do? Thank you for any suggestions. -Michael ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] RGW: Reshard index of non-master zones in multi-site
On Tue, 19 Feb 2019 at 10:11, Iain Buclaw wrote: > > > # ./radosgw-gc-bucket-indexes.sh master.rgw.buckets.index | wc -l > 7511 > > # ./radosgw-gc-bucket-indexes.sh secondary1.rgw.buckets.index | wc -l > 3509 > > # ./radosgw-gc-bucket-indexes.sh secondary2.rgw.buckets.index | wc -l > 3801 > Documentation is a horrid mess around the subject on multi-site resharding http://docs.ceph.com/docs/mimic/radosgw/dynamicresharding/#manual-bucket-resharding https://www.suse.com/documentation/suse-enterprise-storage-5/book_storage_admin/data/ogw_bucket_sharding.html (Manual Resharding) https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/html-single/object_gateway_guide_for_red_hat_enterprise_linux/index#manually-resharding-buckets-with-multisite-rgw All disagree with each other over the correct process to reshard indexes in multi-site. Worse, none of them seem to work correctly anyway. Changelog of 13.2.5 looked promising up until the sentence: "These commands should not be used on a multisite setup as the stale instances may be unlikely to be from a reshard and can have consequences". http://docs.ceph.com/docs/master/releases/mimic/#v13-2-5-mimic -- Iain Buclaw *(p < e ? p++ : p) = (c & 0x0f) + '0'; ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] RGW: Reshard index of non-master zones in multi-site
On Tue, 19 Feb 2019 at 10:11, Iain Buclaw wrote:
>
> On Tue, 19 Feb 2019 at 10:05, Iain Buclaw wrote:
> >
> > On Tue, 19 Feb 2019 at 09:59, Iain Buclaw wrote:
> > >
> > > On Wed, 6 Feb 2019 at 09:28, Iain Buclaw wrote:
> > > >
> > > > On Tue, 5 Feb 2019 at 10:04, Iain Buclaw wrote:
> > > > >
> > > > > On Tue, 5 Feb 2019 at 09:46, Iain Buclaw wrote:
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Following the update of one secondary site from 12.2.8 to 12.2.11,
> > > > > > the
> > > > > > following warning have come up.
> > > > > >
> > > > > > HEALTH_WARN 1 large omap objects
> > > > > > LARGE_OMAP_OBJECTS 1 large omap objects
> > > > > > 1 large objects found in pool '.rgw.buckets.index'
> > > > > > Search the cluster log for 'Large omap object found' for more
> > > > > > details.
> > > > > >
> > > > >
> > > > > [...]
> > > > >
> > > > > > Is this the reason why resharding hasn't propagated?
> > > > > >
> > > > >
> > > > > Furthermore, infact it looks like the index is broken on the
> > > > > secondaries.
> > > > >
> > > > > On the master:
> > > > >
> > > > > # radosgw-admin bi get --bucket=mybucket --object=myobject
> > > > > {
> > > > > "type": "plain",
> > > > > "idx": "myobject",
> > > > > "entry": {
> > > > > "name": "myobject",
> > > > > "instance": "",
> > > > > "ver": {
> > > > > "pool": 28,
> > > > > "epoch": 8848
> > > > > },
> > > > > "locator": "",
> > > > > "exists": "true",
> > > > > "meta": {
> > > > > "category": 1,
> > > > > "size": 9200,
> > > > > "mtime": "2018-03-27 21:12:56.612172Z",
> > > > > "etag": "c365c324cda944d2c3b687c0785be735",
> > > > > "owner": "mybucket",
> > > > > "owner_display_name": "Bucket User",
> > > > > "content_type": "application/octet-stream",
> > > > > "accounted_size": 9194,
> > > > > "user_data": ""
> > > > > },
> > > > > "tag": "0ef1a91a-4aee-427e-bdf8-30589abb2d3e.36603989.137292",
> > > > > "flags": 0,
> > > > > "pending_map": [],
> > > > > "versioned_epoch": 0
> > > > > }
> > > > > }
> > > > >
> > > > >
> > > > > On the secondaries:
> > > > >
> > > > > # radosgw-admin bi get --bucket=mybucket --object=myobject
> > > > > ERROR: bi_get(): (2) No such file or directory
> > > > >
> > > > > How does one go about rectifying this mess?
> > > > >
> > > >
> > > > Random blog in language I don't understand seems to allude to using
> > > > radosgw-admin bi put to restore backed up indexes, but not under what
> > > > circumstances you would use such a command.
> > > >
> > > > https://cloud.tencent.com/developer/article/1032854
> > > >
> > > > Would this be safe to run on secondaries?
> > > >
> > >
> > > Removed the bucket on the secondaries and scheduled new sync. However
> > > this gets stuck at some point and radosgw is complaining about:
> > >
> > > data sync: WARNING: skipping data log entry for missing bucket
> > > mybucket:0ef1a91a-4aee-427e-bdf8-30589abb2d3e.92151615.1:21
> > >
> > > Hopeless that RGW can't even do a simple job right, I removed the
> > > problematic bucket on the master, but now there are now hundreds of
> > > shard objects inside the index pool, all look to be orphaned, and
> > > still the warnings for missing bucket continue to happen on the
> > > secondaries. In some cases there's an object on the secondary that
> > > doesn't exist on the master.
> > >
> > > All the while, ceph is still complaining about large omap files.
> > >
> > > $ ceph daemon mon.ceph-mon-1 config get
> > > osd_deep_scrub_large_omap_object_value_sum_threshold
> > > {
> > > "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824"
> > > }
> > >
> > > It seems implausible that the cluster is still complaining about this
> > > when the largest omap contains 71405 entries.
> > >
> > >
> > > I can't run bi purge or metadata rm on the unreferenced entries
> > > because the bucket itself is no more. Can I remove objects from the
> > > index pool using 'rados rm' ?
> > >
> >
> > Possibly related
> >
> > http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-November/031350.html
> >
>
> # ./radosgw-gc-bucket-indexes.sh master.rgw.buckets.index | wc -l
> 7511
>
> # ./radosgw-gc-bucket-indexes.sh secondary1.rgw.buckets.index | wc -l
> 3509
>
> # ./radosgw-gc-bucket-indexes.sh secondary2.rgw.buckets.index | wc -l
> 3801
>
I guess no one uses multisite RadosGW in production then, I'm
absolutely flabbergasted that RGW can't do what should be a simple job
without breaking.
I'm sure I'll have better luck writing a synchronisation tool myself,
or using minio+syncthing.
--
Iain Buclaw
*(p < e ? p++ : p) = (c & 0x0f) + '0';
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
