Hi Gregory, indeed - I still have warnings about 20% free space on CS3 server, where MON lives...strange is that I don't get these warnings with prolonged "ceph -w" output... [root@cs2 ~]# ceph health detail HEALTH_WARN mon.cs3 addr 10.44.xxx.12:6789/0 has 20% avail disk space -- low disk space!
I don't understand, how is this possible to get warnings - I have folowing in each ceph.conf file, under the general section: mon data avail warn = 15 mon data avail crit = 5 I found this settings on ceph mailing list... Thanks a lot, Andrija On 17 June 2014 19:22, Gregory Farnum <[email protected]> wrote: > Try running "ceph health detail" on each of the monitors. Your disk space > thresholds probably aren't configured correctly or something. > -Greg > > Software Engineer #42 @ http://inktank.com | http://ceph.com > > > On Tue, Jun 17, 2014 at 2:09 AM, Andrija Panic <[email protected]> > wrote: > >> Hi, >> >> thanks for that, but is not space issue: >> >> OSD drives are only 12% full. >> and /var drive on which MON lives is over 70% only on CS3 server, but I >> have increased alert treshold in ceph.conf (mon data avail warn = 15, mon >> data avail crit = 5), and since I increased them those alerts are gone >> (anyway, these alerts for /var full over 70% can be normally seen in logs >> and in ceph -w output). >> >> Here I get no normal/visible warning in eather logs or ceph -w output... >> >> Thanks, >> Andrija >> >> >> >> >> On 17 June 2014 11:00, Stanislav Yanchev <[email protected]> wrote: >> >>> Try grep in cs1 and cs3 could be a disk space issue. >>> >>> >>> >>> >>> >>> Regards, >>> >>> *Stanislav Yanchev* >>> Core System Administrator >>> >>> [image: MAX TELECOM] >>> >>> Mobile: +359 882 549 441 >>> [email protected] >>> www.maxtelecom.bg >>> >>> >>> *From:* ceph-users [mailto:[email protected]] *On >>> Behalf Of *Andrija Panic >>> *Sent:* Tuesday, June 17, 2014 11:57 AM >>> *To:* Christian Balzer >>> *Cc:* [email protected] >>> *Subject:* Re: [ceph-users] Cluster status reported wrongly as >>> HEALTH_WARN >>> >>> >>> >>> Hi Christian, >>> >>> >>> >>> that seems true, thanks. >>> >>> >>> >>> But again, there are only occurence in GZ logs files (that were >>> logrotated, not in current log files): >>> >>> Example: >>> >>> >>> >>> [root@cs2 ~]# grep -ir "WRN" /var/log/ceph/ >>> >>> Binary file /var/log/ceph/ceph-mon.cs2.log-20140612.gz matches >>> >>> Binary file /var/log/ceph/ceph.log-20140614.gz matches >>> >>> Binary file /var/log/ceph/ceph.log-20140611.gz matches >>> >>> Binary file /var/log/ceph/ceph.log-20140612.gz matches >>> >>> Binary file /var/log/ceph/ceph.log-20140613.gz matches >>> >>> >>> >>> Thanks, >>> >>> Andrija >>> >>> >>> >>> On 17 June 2014 10:48, Christian Balzer <[email protected]> wrote: >>> >>> >>> Hello, >>> >>> >>> On Tue, 17 Jun 2014 10:30:44 +0200 Andrija Panic wrote: >>> >>> > Hi, >>> > >>> > I have 3 node (2 OSD per node) CEPH cluster, running fine, not much >>> data, >>> > network also fine: >>> > Ceph ceph-0.72.2. >>> > >>> > When I issue "ceph status" command, I get randomly HEALTH_OK, and >>> > imidiately after that when repeating command, I get HEALTH_WARN >>> > >>> > Examle given down - these commands were issues within less than 1 sec >>> > between them >>> > There are NO occuring of word "warn" in the logs (grep -ir "warn" >>> > /var/log/ceph) on any of the servers... >>> > I get false alerts with my status monitoring script, for this reason... >>> > >>> >>> If I recall correctly, the logs will show INF, WRN and ERR, so grep for >>> WRN. >>> >>> Regards, >>> >>> Christian >>> >>> >>> > Any help would be greatly appriciated. >>> > >>> > Thanks, >>> > >>> > [root@cs3 ~]# ceph status >>> > cluster cab20370-bf6a-4589-8010-8d5fc8682eab >>> > health HEALTH_OK >>> > monmap e2: 3 mons at >>> > >>> {cs1=10.44.xxx.10:6789/0,cs2=10.44.xxx.11:6789/0,cs3=10.44.xxx.12:6789/0}, >>> > election epoch 122, quorum 0,1,2 cs1,cs2,cs3 >>> > osdmap e890: 6 osds: 6 up, 6 in >>> > pgmap v2379904: 448 pgs, 4 pools, 862 GB data, 217 kobjects >>> > 2576 GB used, 19732 GB / 22309 GB avail >>> > 448 active+clean >>> > client io 17331 kB/s rd, 113 kB/s wr, 176 op/s >>> > >>> > [root@cs3 ~]# ceph status >>> > cluster cab20370-bf6a-4589-8010-8d5fc8682eab >>> > health HEALTH_WARN >>> > monmap e2: 3 mons at >>> > >>> {cs1=10.44.xxx.10:6789/0,cs2=10.44.xxx.11:6789/0,cs3=10.44.xxx.12:6789/0}, >>> > election epoch 122, quorum 0,1,2 cs1,cs2,cs3 >>> > osdmap e890: 6 osds: 6 up, 6 in >>> > pgmap v2379905: 448 pgs, 4 pools, 862 GB data, 217 kobjects >>> > 2576 GB used, 19732 GB / 22309 GB avail >>> > 448 active+clean >>> > client io 28383 kB/s rd, 566 kB/s wr, 321 op/s >>> > >>> > [root@cs3 ~]# ceph status >>> > cluster cab20370-bf6a-4589-8010-8d5fc8682eab >>> > health HEALTH_OK >>> > monmap e2: 3 mons at >>> > >>> {cs1=10.44.xxx.10:6789/0,cs2=10.44.xxx.11:6789/0,cs3=10.44.xxx.12:6789/0}, >>> > election epoch 122, quorum 0,1,2 cs1,cs2,cs3 >>> > osdmap e890: 6 osds: 6 up, 6 in >>> > pgmap v2379913: 448 pgs, 4 pools, 862 GB data, 217 kobjects >>> > 2576 GB used, 19732 GB / 22309 GB avail >>> > 448 active+clean >>> > client io 21632 kB/s rd, 49354 B/s wr, 283 op/s >>> > >>> >>> >>> -- >>> >>> Christian Balzer Network/Systems Engineer >>> [email protected] Global OnLine Japan/Fusion Communications >>> http://www.gol.com/ >>> >>> >>> >>> >>> >>> -- >>> >>> >>> >>> Andrija Panić >>> >>> -------------------------------------- >>> >>> http://admintweets.com >>> >>> -------------------------------------- >>> >>> <http://gfidisc.maxtelecom.bg> >>> >>> *Confidentiality notice* >>> ------------------------------ >>> >>> >>> >>> The information contained in this message (including any attachments) is >>> confidential and may be legally privileged or otherwise protected from >>> disclosure. This message is intended solely for the addressee(s). If you >>> are not the intended recipient, please notify the sender by return e-mail >>> and delete this message from your system. Any unauthorised use, >>> reproduction, or dissemination of this message is strictly prohibited. Any >>> liability arising from any third party acting, or refraining from acting, >>> on any information contained in this e-mail is hereby excluded. Please note >>> that e-mails are susceptible to change. Max Telecom shall not be liable for >>> the improper or incomplete transmission of the information contained in >>> this communication, nor shall it be liable for any delay in its receipt. >>> >>> <http://gfidisc.maxtelecom.bg> >>> >> >> >> >> -- >> >> Andrija Panić >> -------------------------------------- >> http://admintweets.com >> -------------------------------------- >> >> _______________________________________________ >> ceph-users mailing list >> [email protected] >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> > -- Andrija Panić -------------------------------------- http://admintweets.com --------------------------------------
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
