I don't know if this will cause the problem, but I do have a couple of
recommendations.
You don't need 10 monitors for a 10 node cluster. 3 monitors should be
fine. More monitors causes more overhead for the monitors to sync cluster
state. The most I've heard of is somebody using 7 monitors. I believe
that setup had 1 monitor in each row, with 7 rows, for a total of hundreds
of nodes in the cluster. 10 should work, but it's overkill for your setup.
You shouldn't name your OSDs like that. The OSDs allocate an array
containing MAX_OSD_ID entries. By naming your OSDs 101 to 1001, you're
making each OSD allocate an array 1000 entries long, instead of 50 entries
long. You should use sequential IDs, created by ceph osd create. Use ceph
osd tree to lookup which host has an osd.
I don't know if these changes will fix your issue, but the OSD names might
be causing it.
On Wed, Jun 18, 2014 at 5:06 PM, wsnote <wsn...@163.com> wrote:
> It happeds when I have installed ceph cluster and start it. I have not
> done anything in it.
> I use one moniter per server. That is to say, there are 10 moniters in the
> ceph cluster.
> I tried to reduce the number of moniters, but the problem remained.
>
> I give the info about some commands.
>
> 1. command: ceph -s
> cluster 084265e5-1dc0-46ca-912b-81b9b0127461
> health HEALTH_WARN 374 pgs degraded; 11053 pgs down; 6805 pgs
> incomplete; 51729 pgs peering; 62838 pgs stale; 187069 pgs stuck inactive;
> 62838 pgs stuck stale; 187153 pgs stuck unclean; 45 requests are blocked >
> 32 sec; 1/15 in osds are down
> monmap e1: 10 mons at {1=
> 122.228.248.168:6789/0,10=122.228.248.177:6789/0,2=122.228.248.169:6789/0,3=122.228.248.170:6789/0,4=122.228.248.171:6789/0,5=122.228.248.172:6789/0,6=122.228.248.173:6789/0,7=122.228.248.174:6789/0,8=122.228.248.175:6789/0,9=122.228.248.176:6789/0},
> election epoch 110, quorum 0,1,2,3,4,5,6,7,8,9 1,2,3,4,5,6,7,8,9,10
> mdsmap e46: 1/1/1 up {0=1=up:creating}, 9 up:standby
> osdmap e786: 50 osds: 14 up, 15 in
> pgmap v1293: 193152 pgs, 3 pools, 0 bytes data, 0 objects
> 16997 MB used, 44673 GB / 44690 GB avail
> 122757 creating
> 2 stale+active+degraded+remapped
> 6480 peering
> 6496 stale+incomplete
> 89 stale+replay+degraded
> 10022 stale+down+peering
> 5 stale+active+replay+degraded
> 152 stale+remapped
> 1 stale+active+remapped
> 293 incomplete
> 1858 stale+remapped+peering
> 3802 stale+replay
> 774 down+peering
> 201 stale+degraded
> 3892 stale+active+clean+replay
> 76 stale+active+degraded
> 10 remapped+peering
> 16 stale+remapped+incomplete
> 1525 stale
> 1 stale+replay+degraded+remapped
> 257 stale+down+remapped+peering
> 2107 stale+active+clean
> 32328 stale+peering
> 8 stale+replay+remapped
>
> 2. command: ceph osd tree
> # id weight type name up/down reweight
> -1 50 root default
> -3 50 rack unknownrack
> -2 5 host yun177
> 1001 1 osd.1001 down 0
> 1002 1 osd.1002 down 0
> 1003 1 osd.1003 down 0
> 1004 1 osd.1004 down 0
> 1005 1 osd.1005 down 0
> -4 5 host yun168
> 101 1 osd.101 down 0
> 102 1 osd.102 down 0
> 103 1 osd.103 down 0
> 104 1 osd.104 down 0
> 105 1 osd.105 down 0
> -5 5 host yun169
> 201 1 osd.201 down 0
> 202 1 osd.202 down 0
> 203 1 osd.203 down 1
> 204 1 osd.204 up 1
> 205 1 osd.205 down 0
> -6 5 host yun170
> 301 1 osd.301 down 0
> 302 1 osd.302 down 0
> 303 1 osd.303 up 1
> 304 1 osd.304 down 0
> 305 1 osd.305 down 0
> -7 5 host yun171
> 401 1 osd.401 down 0
> 402 1 osd.402 down 0
> 403 1 osd.403 up 1
> 404 1 osd.404 up 1
> 405 1 osd.405 down 0
> -8 5 host yun172
> 501 1 osd.501 down 0
> 502 1 osd.502 down 0
> 503 1 osd.503 down 0
> 504 1 osd.504 down 0
> 505 1 osd.505 up 1
> -9 5 host yun173
> 601 1 osd.601 down 0
> 602 1 osd.602 down 0
> 603 1 osd.603 down 0
> 604 1 osd.604 down 0
> 605 1 osd.605 down 0
> -10 5 host yun174
> 701 1 osd.701 down 0
> 702 1 osd.702 down 0
> 703 1 osd.703 down 0
> 704 1 osd.704 down 0
> 705 1 osd.705 down 0
> -11 5 host yun175
> 801 1 osd.801 down 0
> 802 1 osd.802 up 1
> 803 1 osd.803 up 1
> 804 1 osd.804 up 1
> 805 1 osd.805 up 1
> -12 5 host yun176
> 901 1 osd.901 up 1
> 902 1 osd.902 up 1
> 903 1 osd.903 up 1
> 904 1 osd.904 up 1
> 905 1 osd.905 up 1
>
> 3. command: ceph health detail
> HEALTH_WARN 374 pgs degraded; 11053 pgs down; 6805 pgs incomplete; 51729
> pgs peering; 62838 pgs stale; 187069 pgs stuck inactive; 62838 pgs stuck
> stale; 187153 pgs stuck unclean; 45 requests are blocked > 32 sec; 10 osds
> have slow requests; 1/15 in osds are down
> pg 0.fb7f is stuck inactive since forever, current state down+peering,
> last acting [404]
> pg 1.fb7e is stuck inactive since forever, current state
> stale+down+peering, last acting [202]
> pg 2.fb7d is stuck inactive since forever, current state creating, last
> acting [204,902,505]
> pg 0.fb7e is stuck inactive since forever, current state stale+peering,
> last acting [704,1004,504]
> pg 1.fb7f is stuck inactive since forever, current state creating, last
> acting [404,204,804]
> pg 2.fb7c is stuck inactive since forever, current state creating, last
> acting [805,901,404]
> pg 0.fb7d is stuck inactive since forever, current state creating, last
> acting [903,204,803]
> pg 1.fb7c is stuck inactive since forever, current state creating, last
> acting [802,905,505]
> pg 2.fb7f is stuck inactive since forever, current state creating, last
> acting [804,902]
> pg 1.fb7d is stuck inactive since forever, current state creating, last
> acting [803,404,204]
> pg 2.fb7e is stuck inactive since forever, current state creating, last
> acting [404,905,802]
> pg 0.fb7c is stuck inactive since forever, current state stale+peering,
> last acting [904,405]
> pg 2.fb79 is stuck inactive since forever, current state creating, last
> acting [804,901]
> pg 1.fb7a is stuck inactive since forever, current state creating, last
> acting [903,505,803]
> pg 0.fb7b is stuck inactive since forever, current state stale+peering,
> last acting [801,503]
> pg 2.fb78 is stuck inactive since forever, current state creating, last
> acting [903,404]
> pg 1.fb7b is stuck inactive since forever, current state creating, last
> acting [505,803,303]
> pg 0.fb7a is stuck inactive since forever, current state
> stale+remapped+peering, last acting [1003]
> pg 2.fb7b is stuck inactive since forever, current state creating, last
> acting [303,505,802]
> pg 1.fb78 is stuck inactive since forever, current state creating, last
> acting [803,303,905]
> pg 0.fb79 is stuck inactive since forever, current state creating, last
> acting [901,403,805]
> pg 0.fb78 is stuck inactive since forever, current state creating, last
> acting [404,901]
> pg 2.fb7a is stuck inactive since forever, current state creating, last
> acting [403,303,805]
> pg 1.fb79 is stuck inactive since forever, current state creating, last
> acting [803,901,404]
> pg 0.fb77 is stuck inactive for 24155.756030, current state stale+peering,
> last acting [101,1005]
> pg 1.fb76 is stuck inactive since forever, current state creating, last
> acting [901,505,403]
> pg 2.fb75 is stuck inactive since forever, current state creating, last
> acting [905,403,204]
> pg 1.fb77 is stuck inactive since forever, current state creating, last
> acting [905,805,204]
> pg 2.fb74 is stuck inactive since forever, current state creating, last
> acting [901,404]
> pg 0.fb75 is stuck inactive since forever, current state creating, last
> acting [903,403]
> pg 1.fb74 is stuck inactive since forever, current state creating, last
> acting [901,204,403]
> pg 2.fb77 is stuck inactive since forever, current state creating, last
> acting [505,802,905]
> pg 0.fb74 is stuck inactive for 24042.660267, current state
> stale+incomplete, last acting [101]
> pg 1.fb75 is stuck inactive since forever, current state creating, last
> acting [905,403,804]
>
>
>
>
> At 2014-06-19 04:31:09,"Craig Lewis" <cle...@centraldesktop.com> wrote:
>
> I haven't seen behavior like that. I have seen my OSDs use a lot of RAM
> while they're doing a recovery, but it goes back down when they're done.
>
> Your OSD is doing something, it's using 126% CPU. What does `ceph osd
> tree` and `ceph health detail` say?
>
>
> When you say you're installing Ceph on 10 severs, are you running a
> monitor on all 10 servers?
>
>
>
>
> On Wed, Jun 18, 2014 at 4:18 AM, wsnote <wsn...@163.com> wrote:
>
>> If I install ceph in 10 servers with one disk each servers, the problem
>> remains.
>> This is the memory usage of ceph-osd.
>> ceph-osd VIRT:10.2G, RES: 4.2G
>> The usage of ceph-osd is too big!
>>
>>
>> At 2014-06-18 16:51:02,wsnote <wsn...@163.com> wrote:
>>
>> Hi, Lewis!
>> I come up with a question and don't know how to solve, so I ask you for
>> help.
>> I can succeed to install ceph in a cluster with 3 or 4 servers but fail
>> to do it with 10 servers.
>> I install it and start it, then there would be a server whose memory
>> rises up to 100% and this server crash.I have to restart it.
>> All the config are the same.I don't know what's the problem.
>> Can you give some suggestion?
>> Thanks!
>>
>> ceph.conf:
>> [global]
>> auth supported = none
>>
>> ;auth_service_required = cephx
>> ;auth_client_required = cephx
>> ;auth_cluster_required = cephx
>> filestore_xattr_use_omap = true
>>
>> max open files = 131072
>> log file = /var/log/ceph/$name.log
>> pid file = /var/run/ceph/$name.pid
>> keyring = /etc/ceph/keyring.admin
>>
>> ;mon_clock_drift_allowed = 1 ;clock skew detected
>>
>> [mon]
>> mon data = /data/mon$id
>> keyring = /etc/ceph/keyring.$name
>> [mds]
>> mds data = /data/mds$id
>> keyring = /etc/ceph/keyring.$name
>> [osd]
>> osd data = /data/osd$id
>> osd journal = /data/osd$id/journal
>> osd journal size = 1024
>> keyring = /etc/ceph/keyring.$name
>> osd mkfs type = xfs
>> osd mount options xfs = rw,noatime
>> osd mkfs options xfs = -f
>> filestore fiemap = false
>>
>> In every server, there is an mds, an mon, 11 osd with 4TB space each.
>> mon address is public IP, and osd address has an public IP and an cluster
>> IP.
>>
>> wsnote
>>
>>
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
