Re: [ceph-users] ceph; pg scrub errors
On Tue, Sep 24, 2019 at 10:51 PM M Ranga Swami Reddy
wrote:
>
> Interestingly - "rados list-inconsistent-obj ${PG} --format=json" not
> showing any objects inconsistent-obj.
> And also "rados list-missing-obj ${PG} --format=json" also not showing any
> missing or unfound objects.
Complete a scrub of ${PG} just before you run these commands.
>
> Thanks
> Swami
>
> On Mon, Sep 23, 2019 at 8:18 PM Robert LeBlanc wrote:
>>
>> On Thu, Sep 19, 2019 at 4:34 AM M Ranga Swami Reddy
>> wrote:
>> >
>> > Hi-Iam using ceph 12.2.11. here I am getting a few scrub errors. To fix
>> > these scrub error I ran the "ceph pg repair ".
>> > But scrub error not going and the repair is talking long time like 8-12
>> > hours.
>>
>> Depending on the size of the PGs and how active the cluster is, it
>> could take a long time as it takes another deep scrub to happen to
>> clear the error status after a repair. Since it is not going away,
>> either the problem is too complicated to automatically repair and
>> needs to be done by hand, or the problem is repaired and when it
>> deep-scrubs to check it, the problem has reappeared or another problem
>> was found and the disk needs to be replaced.
>>
>> Try running:
>> rados list-inconsistent-obj ${PG} --format=json
>>
>> and see what the exact problems are.
>>
>> Robert LeBlanc
>> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Cheers,
Brad
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] cephfs performance issue MDSs report slow requests and osd memory usage
On Tue, Sep 24, 2019 at 4:33 AM Thomas <74cmo...@gmail.com> wrote: > > Hi, > > I'm experiencing the same issue with this setting in ceph.conf: > osd op queue = wpq > osd op queue cut off = high > > Furthermore I cannot read any old data in the relevant pool that is > serving CephFS. > However, I can write new data and read this new data. If you restarted all the OSDs with this setting, it won't necessarily prevent any blocked IO, it just really helps prevent the really long blocked IO and makes sure that IO is eventually done in a more fair manner. It sounds like you may have some MDS issues that are deeper than my understanding. First thing I'd try is to bounce the MDS service. > > If I want to add this my ceph-ansible playbook parameters, in which files I > > should add it and what is the best way to do it ? > > > > Add those 3 lines in all.yml or osds.yml ? > > > > ceph_conf_overrides: > > global: > > osd_op_queue_cut_off: high > > > > Is there another (better?) way to do that? I can't speak to either of those approaches. I wanted all my config in a single file, so I put it in my inventory file, but it looks like you have the right idea. Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Nautilus : ceph dashboard ssl not working
On 9/24/19 1:37 PM, Miha Verlic wrote: > I've got slightly different problem. After a few days of running fine, > dashboard stops working because it is apparently seeking for wrong > certificate file in /tmp. If I restart ceph-mgr it starts to work again. Does the restart trigger the creation of a similar-looking file in /tmp? I wonder if there's some kind of cron job that cleans up the /tmp directory every now and then... Lenz -- SUSE Software Solutions Germany GmbH - Maxfeldstr. 5 - 90409 Nuernberg GF: Felix Imendörffer, HRB 247165 (AG Nürnberg) signature.asc Description: OpenPGP digital signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph; pg scrub errors
Interestingly - "rados list-inconsistent-obj ${PG} --format=json" not
showing any objects inconsistent-obj.
And also "rados list-missing-obj ${PG} --format=json" also not showing any
missing or unfound objects.
Thanks
Swami
On Mon, Sep 23, 2019 at 8:18 PM Robert LeBlanc wrote:
> On Thu, Sep 19, 2019 at 4:34 AM M Ranga Swami Reddy
> wrote:
> >
> > Hi-Iam using ceph 12.2.11. here I am getting a few scrub errors. To fix
> these scrub error I ran the "ceph pg repair ".
> > But scrub error not going and the repair is talking long time like 8-12
> hours.
>
> Depending on the size of the PGs and how active the cluster is, it
> could take a long time as it takes another deep scrub to happen to
> clear the error status after a repair. Since it is not going away,
> either the problem is too complicated to automatically repair and
> needs to be done by hand, or the problem is repaired and when it
> deep-scrubs to check it, the problem has reappeared or another problem
> was found and the disk needs to be replaced.
>
> Try running:
> rados list-inconsistent-obj ${PG} --format=json
>
> and see what the exact problems are.
>
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Nautilus : ceph dashboard ssl not working
On 17. 09. 19 22:49, Michel Raabe wrote:
> ceph config-key dump | grep dashboard/[crt,key]
>
> Try this:
>
> ceph config-key set mgr mgr/dashboard/crt -i ssl.crt
> ceph config-key set mgr mgr/dashboard/key -i ssl.key
I've got slightly different problem. After a few days of running fine,
dashboard stops working because it is apparently seeking for wrong
certificate file in /tmp. If I restart ceph-mgr it starts to work again.
Sep 24 13:26:26 cephtest01 ceph-mgr[2247]: [24/Sep/2019:13:26:26] ENGINE
Error in HTTPServer.tick
Sep 24 13:26:26 cephtest01 ceph-mgr[2247]: Traceback (most recent call
last):
Sep 24 13:26:26 cephtest01 ceph-mgr[2247]: File
"/usr/lib/python2.7/site-packages/cherrypy/wsgiserver/wsgiserver2.py",
line 1837, in start
Sep 24 13:26:26 cephtest01 ceph-mgr[2247]: self.tick()
Sep 24 13:26:26 cephtest01 ceph-mgr[2247]: File
"/usr/lib/python2.7/site-packages/cherrypy/wsgiserver/wsgiserver2.py",
line 1902, in tick
Sep 24 13:26:26 cephtest01 ceph-mgr[2247]: s, ssl_env =
self.ssl_adapter.wrap(s)
Sep 24 13:26:26 cephtest01 ceph-mgr[2247]: File
"/usr/lib/python2.7/site-packages/cherrypy/wsgiserver/ssl_builtin.py",
line 52, in wrap
Sep 24 13:26:26 cephtest01 ceph-mgr[2247]: keyfile=self.private_key,
ssl_version=ssl.PROTOCOL_SSLv23)
Sep 24 13:26:26 cephtest01 ceph-mgr[2247]: File
"/usr/lib64/python2.7/ssl.py", line 934, in wrap_socket
Sep 24 13:26:26 cephtest01 ceph-mgr[2247]: ciphers=ciphers)
Sep 24 13:26:26 cephtest01 ceph-mgr[2247]: File
"/usr/lib64/python2.7/ssl.py", line 547, in __init__
Sep 24 13:26:26 cephtest01 ceph-mgr[2247]:
self._context.load_cert_chain(certfile, keyfile)
Sep 24 13:26:26 cephtest01 ceph-mgr[2247]: IOError: [Errno 2] No such
file or directory
[pid 155620] open("/tmp/tmpM0BDKU", O_RDONLY) = -1 ENOENT (No such file
or directory)
In tmp folder, there is no tmpM0BDKU. Both crt and key exist under a
different name, created when mgr daemon was started:
~]# ls -la
/tmp/systemd-private-a17bab1bcad649cc89c8c9f087497236-ceph-mgr@cephtest01.service-vtiIiy/tmp/
total 8
drwxrwxrwt 2 root root 40 Sep 19 12:54 .
drwx-- 3 root root 17 Sep 19 10:24 ..
-rw--- 1 ceph ceph 2435 Sep 19 12:54 tmpHeqGFS
-rw--- 1 ceph ceph 3268 Sep 19 12:54 tmpNSoREs
I'm running latest version on fully updated CentOS 7.7:
"mgr": {
"ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba)
nautilus (stable)": 3
},
--
Miha
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] cephfs performance issue MDSs report slow requests and osd memory usage
Hi, I'm experiencing the same issue with this setting in ceph.conf: osd op queue = wpq osd op queue cut off = high Furthermore I cannot read any old data in the relevant pool that is serving CephFS. However, I can write new data and read this new data. Regards Thomas Am 24.09.2019 um 10:24 schrieb Yoann Moulin: > Hello, > >>> I have a Ceph Nautilus Cluster 14.2.1 for cephfs only on 40x 1.8T SAS disk >>> (no SSD) in 20 servers. >>> >>> I often get "MDSs report slow requests" and plenty of "[WRN] 3 slow >>> requests, 0 included below; oldest blocked for > 60281.199503 secs" >>> >>> After a few investigations, I saw that ALL ceph-osd process eat a lot of >>> memory, up to 130GB RSS each. It this value normal? May this related to >>> slow requests? Is disk only increasing the probability to get slow requests? >> If you haven't set: >> >> osd op queue cut off = high >> >> in /etc/ceph/ceph.conf on your OSDs, I'd give that a try. It should >> help quite a bit with pure HDD clusters. > OK I'll try this, thanks. > > If I want to add this my ceph-ansible playbook parameters, in which files I > should add it and what is the best way to do it ? > > Add those 3 lines in all.yml or osds.yml ? > > ceph_conf_overrides: > global: > osd_op_queue_cut_off: high > > Is there another (better?) way to do that? > > Thanks for your help. > > Best regards, > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Nautilus : ceph dashboard ssl not working
Thank you Ricardo Dias On Tue, Sep 17, 2019 at 2:13 PM Ricardo Dias wrote: > Hi Muthu, > > The command you used is only available in v14.2.3. To set the ssl > certificate in v14.2.2 you need to use the following commands: > > $ ceph config-key set mgr/dashboard/crt -i dashboard.crt > $ ceph config-key set mgr/dashboard/key -i dashboard.key > > The above commands will emit a deprecation warning that you can ignore. > > Thanks, > Ricardo Dias > > > From: ceph-users on behalf of nokia > ceph > Sent: Monday, September 16, 2019 10:30 > To: Ceph Users > Subject: [ceph-users] Nautilus : ceph dashboard ssl not working > > Hi Team, > In ceph 14.2.2 , ceph dashboard does not have set-ssl-certificate . > We are trying to enable ceph dashboard and while using the ssl certificate > and key , it is not working . > > cn5.chn5au1c1.cdn ~# ceph dashboard set-ssl-certificate -i dashboard.crt > no valid command found; 10 closest matches: > dashboard set-grafana-update-dashboards > dashboard reset-prometheus-api-host > dashboard reset-ganesha-clusters-rados-pool-namespace > dashboard set-grafana-api-username > dashboard get-audit-api-log-payload > dashboard get-grafana-api-password > dashboard get-grafana-api-username > dashboard set-rgw-api-access-key > dashboard reset-rgw-api-host > dashboard set-prometheus-api-host > Error EINVAL: invalid command > cn5.chn5au1c1.cdn ~# ceph -v > ceph version 14.2.2 (4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus > (stable) > > How to set crt and key in this case. > > Thanks, > Muthu > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] cephfs performance issue MDSs report slow requests and osd memory usage
Hello, >> I have a Ceph Nautilus Cluster 14.2.1 for cephfs only on 40x 1.8T SAS disk >> (no SSD) in 20 servers. >> >> I often get "MDSs report slow requests" and plenty of "[WRN] 3 slow >> requests, 0 included below; oldest blocked for > 60281.199503 secs" >> >> After a few investigations, I saw that ALL ceph-osd process eat a lot of >> memory, up to 130GB RSS each. It this value normal? May this related to >> slow requests? Is disk only increasing the probability to get slow requests? > > If you haven't set: > > osd op queue cut off = high > > in /etc/ceph/ceph.conf on your OSDs, I'd give that a try. It should > help quite a bit with pure HDD clusters. OK I'll try this, thanks. If I want to add this my ceph-ansible playbook parameters, in which files I should add it and what is the best way to do it ? Add those 3 lines in all.yml or osds.yml ? ceph_conf_overrides: global: osd_op_queue_cut_off: high Is there another (better?) way to do that? Thanks for your help. Best regards, -- Yoann Moulin EPFL IC-IT ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] eu.ceph.com mirror out of sync?
Dear Wido, On 2019-09-24 08:53, Wido den Hollander wrote: On 9/17/19 11:01 PM, Oliver Freyermuth wrote: Dear Cephalopodians, I realized just now that: https://eu.ceph.com/rpm-nautilus/el7/x86_64/ still holds only released up to 14.2.2, and nothing is to be seen of 14.2.3 or 14.2.4, while the main repository at: https://download.ceph.com/rpm-nautilus/el7/x86_64/ looks as expected. Is this issue with the eu.ceph.com mirror already knwon? I missed this message and I see what's going on. Going to fix it right away. I manage this mirror. many thanks, it looks like it's already fixed now, at least the new packages are popping up :-). I'll also contact the other mirror owners whose mirrors appear to have issues or are out-of-sync, now that I have been pointed to the list of people managing them. Cheers and thanks, Oliver Wido Cheers, Oliver ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com smime.p7s Description: S/MIME Cryptographic Signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] eu.ceph.com mirror out of sync?
On 9/17/19 11:01 PM, Oliver Freyermuth wrote: > Dear Cephalopodians, > > I realized just now that: > https://eu.ceph.com/rpm-nautilus/el7/x86_64/ > still holds only released up to 14.2.2, and nothing is to be seen of > 14.2.3 or 14.2.4, > while the main repository at: > https://download.ceph.com/rpm-nautilus/el7/x86_64/ > looks as expected. > > Is this issue with the eu.ceph.com mirror already knwon? > I missed this message and I see what's going on. Going to fix it right away. I manage this mirror. Wido > Cheers, > Oliver > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
