Hi Joao,
When I run the following command on the node "ceph --admin-daemon
/var/run/ceph/ceph-mon.dec.asok mon_status"
I get the following output:
{ "name": "dec",
"rank": 0,
"state": "probing",
"election_epoch": 0,
"quorum": [],
"outside_quorum": [
"dec"],
"extra_probe_peers": [
"15.213.24.230:6789\/0",
"15.213.24.231:6789\/0"],
"sync_provider": [],
"monmap": { "epoch": 0,
"fsid": "166ba014-9b5b-4cea-8403-ecdb6e0a1763",
"modified": "0.000000",
"created": "0.000000",
"mons": [
{ "rank": 0,
"name": "dec",
"addr": "15.213.24.241:6789\/0"},
{ "rank": 1,
"name": "jul",
"addr": "0.0.0.0:0\/1"},
{ "rank": 2,
"name": "julilo",
"addr": "0.0.0.0:0\/2"}]}
}
And the addr for the other two nodes is "0.0.0.0" in monitor list. Any
ideas on this??
-Abhay
On Tue, Oct 8, 2013 at 3:15 PM, Abhay Sachan <[email protected]> wrote:
> Hi Joao,
> I gave the path for the keyring, and these messages are being printed on
> the screen:
>
> 2013-10-07 15:18:55.048151 7fa1c43bc700 0 -- :/1026989 >>
> 15.213.24.231:6789/0 pipe(0x7fa1b4000990 sd=4 :0 s=1 pgs=0 cs=0 l=1
> c=0x7fa1b400dfe0).fault
> 2013-10-07 15:18:58.048774 7fa1c42bb700 0 -- :/1026989 >>
> 15.213.24.230:6789/0 pipe(0x7fa1b4001cb0 sd=3 :0 s=1 pgs=0 cs=0 l=1
> c=0x7fa1b4002290).fault
> 2013-10-07 15:19:01.049193 7fa1c43bc700 0 -- :/1026989 >>
> 15.213.24.231:6789/0 pipe(0x7fa1b4000990 sd=4 :0 s=1 pgs=0 cs=0 l=1
> c=0x7fa1b40038b0).fault
> 2013-10-07 15:19:04.049690 7fa1c42bb700 0 -- :/1026989 >>
> 15.213.24.241:6789/0 pipe(0x7fa1b4001cb0 sd=4 :0 s=1 pgs=0 cs=0 l=1
> c=0x7fa1b4002590).fault
> 2013-10-07 15:19:07.050098 7fa1c43bc700 0 -- :/1026989 >>
> 15.213.24.231:6789/0 pipe(0x7fa1b4000990 sd=4 :0 s=1 pgs=0 cs=0 l=1
> c=0x7fa1b40038b0).fault
> 2013-10-07 15:19:10.050467 7fa1c42bb700 0 -- :/1026989 >>
> 15.213.24.230:6789/0 pipe(0x7fa1b4001cb0 sd=4 :0 s=1 pgs=0 cs=0 l=1
> c=0x7fa1b4005710).fault
> 2013-10-07 15:19:13.051033 7fa1c43bc700 0 -- :/1026989 >>
> 15.213.24.241:6789/0 pipe(0x7fa1b4000990 sd=3 :0 s=1 pgs=0 cs=0 l=1
> c=0x7fa1b4005870).fault
> 2013-10-07 15:19:16.051257 7fa1c42bb700 0 -- :/1026989 >>
> 15.213.24.231:6789/0 pipe(0x7fa1b4001cb0 sd=4 :0 s=1 pgs=0 cs=0 l=1
> c=0x7fa1b40042e0).fault
> 2013-10-07 15:19:19.051823 7fa1c43bc700 0 -- :/1026989 >>
> 15.213.24.241:6789/0 pipe(0x7fa1b4000990 sd=3 :0 s=1 pgs=0 cs=0 l=1
> c=0x7fa1b4004990).fault
> 2013-10-07 15:19:22.052177 7fa1c42bb700 0 -- :/1026989 >>
> 15.213.24.230:6789/0 pipe(0x7fa1b4001cb0 sd=4 :0 s=1 pgs=0 cs=0 l=1
> c=0x7fa1b40042e0).fault
> 2013-10-07 15:19:25.052526 7fa1c43bc700 0 -- :/1026989 >>
> 15.213.24.241:6789/0 pipe(0x7fa1b4000990 sd=4 :0 s=1 pgs=0 cs=0 l=1
> c=0x7fa1b400f4a0).fault
> 2013-10-07 15:19:28.053000 7fa1c42bb700 0 -- :/1026989 >>
> 15.213.24.231:6789/0 pipe(0x7fa1b4007d90 sd=4 :0 s=1 pgs=0 cs=0 l=1
> c=0x7fa1b4007250).fault
> 2013-10-07 15:19:31.053452 7fa1c43bc700 0 -- :/1026989 >>
> 15.213.24.230:6789/0 pipe(0x7fa1b4000990 sd=4 :0 s=1 pgs=0 cs=0 l=1
> c=0x7fa1b400f4a0).fault
> 2013-10-07 15:19:34.054070 7fa1c42bb700 0 -- :/1026989 >>
> 15.213.24.241:6789/0 pipe(0x7fa1b4001cb0 sd=3 :0 s=1 pgs=0 cs=0 l=1
> c=0x7fa1b4007d90).fault
> 2013-10-07 15:19:37.054303 7fa1c43bc700 0 -- :/1026989 >>
> 15.213.24.230:6789/0 pipe(0x7fa1b4000990 sd=4 :0 s=1 pgs=0 cs=0 l=1
> c=0x7fa1b400a7e0).fault
> 2013-10-07 15:19:40.054724 7fa1c42bb700 0 -- :/1026989 >>
> 15.213.24.241:6789/0 pipe(0x7fa1b4001cb0 sd=4 :0 s=1 pgs=0 cs=0 l=1
> c=0x7fa1b400a1b0).fault
> 2013-10-07 15:19:43.055099 7fa1c43bc700 0 -- :/1026989 >>
> 15.213.24.231:6789/0 pipe(0x7fa1b4000990 sd=4 :0 s=1 pgs=0 cs=0 l=1
> c=0x7fa1b4009610).fault
> 2013-10-07 15:19:46.055462 7fa1c42bb700 0 -- :/1026989 >>
> 15.213.24.230:6789/0 pipe(0x7fa1b4001cb0 sd=3 :0 s=1 pgs=0 cs=0 l=1
> c=0x7fa1b4007d90).fault
> 2013-10-07 15:19:49.056004 7fa1c43bc700 0 -- :/1026989 >>
> 15.213.24.231:6789/0 pipe(0x7fa1b4000990 sd=4 :0 s=1 pgs=0 cs=0 l=1
> c=0x7fa1b400c5a0).fault
> 2013-10-07 15:19:52.056492 7fa1c42bb700 0 -- :/1026989 >>
> 15.213.24.230:6789/0 pipe(0x7fa1b4001cb0 sd=4 :0 s=1 pgs=0 cs=0 l=1
> c=0x7fa1b400b000).fault
> 2013-10-07 15:19:55.056740 7fa1c43bc700 0 -- :/1026989 >>
> 15.213.24.231:6789/0 pipe(0x7fa1b4000990 sd=4 :0 s=1 pgs=0 cs=0 l=1
> c=0x7fa1b40111b0).fault
> 2013-10-07 15:19:58.057117 7fa1c42bb700 0 -- :/1026989 >>
> 15.213.24.241:6789/0 pipe(0x7fa1b4009110 sd=3 :0 s=1 pgs=0 cs=0 l=1
> c=0x7fa1b400b4a0).fault
> 2013-10-07 15:20:01.057596 7fa1c43bc700 0 -- :/1026989 >>
> 15.213.24.230:6789/0 pipe(0x7fa1b400bc00 sd=4 :0 s=1 pgs=0 cs=0 l=1
> c=0x7fa1b400be60).fault
> 2013-10-07 15:20:04.058191 7fa1c42bb700 0 -- :/1026989 >>
> 15.213.24.231:6789/0 pipe(0x7fa1b400d170 sd=4 :0 s=1 pgs=0 cs=0 l=1
> c=0x7fa1b4015300).fault
> 2013-10-07 15:20:07.058837 7fa1c43bc700 0 -- :/1026989 >>
> 15.213.24.230:6789/0 pipe(0x7fa1b400f640 sd=3 :0 s=1 pgs=0 cs=0 l=1
> c=0x7fa1b400bc00).fault
> 2013-10-07 15:20:10.059147 7fa1c42bb700 0 -- :/1026989 >>
> 15.213.24.231:6789/0 pipe(0x7fa1b400d170 sd=4 :0 s=1 pgs=0 cs=0 l=1
> c=0x7fa1b40015c0).fault
>
>
> Regards,
> Abhay
>
>
>
>
> On Tue, Oct 8, 2013 at 2:44 PM, Joao Eduardo Luis
> <[email protected]>wrote:
>
>> On 08/10/13 05:40, Abhay Sachan wrote:
>>
>>> Hi Joao,
>>> Thanks for replying. All of my monitors are up and running and connected
>>> to each other. "ceph -s" is failing on the cluster with following error:
>>>
>>> 2013-10-07 10:12:25.099261 7fd1b948d700 -1 monclient(hunting): ERROR:
>>> missing keyring, cannot use cephx for authentication
>>> 2013-10-07 10:12:25.099271 7fd1b948d700 0 librados: client.admin
>>> initialization error (2) No such file or directory
>>> Error connecting to cluster: ObjectNotFound
>>>
>>
>> This says it all. You're somehow missing your keyring. Or maybe you're
>> not but you're keeping it in a non-default location and you're not
>> specifying a path to it either on your ceph.conf or via '--keyring <path>'.
>>
>>
>>
>> And the logs on each monitor has lots of entries like this:
>>> NODE 1:
>>>
>>> 2013-10-07 03:58:51.153847 7ff2864c6700 0
>>> mon.jul@0(probing).data_**health(0) update_stats avail 76% total
>>> 42332700
>>> used 7901820 avail 32280480
>>> 2013-10-07 03:59:51.154051 7ff2864c6700 0
>>> mon.jul@0(probing).data_**health(0) update_stats avail 76% total
>>> 42332700
>>> used 7901832 avail 32280468
>>> 2013-10-07 04:00:51.154256 7ff2864c6700 0
>>> mon.jul@0(probing).data_**health(0) update_stats avail 76% total
>>> 42332700
>>> used 7901828 avail 32280472
>>>
>>
>> Those messages are simply reports on the mon store's storage capacity.
>>
>>
>> -Joao
>>
>>
>>>
>>> Regards,
>>> Abhay
>>>
>>> On Thu, Oct 3, 2013 at 8:31 PM, Joao Eduardo Luis <[email protected]
>>> <mailto:[email protected]>**> wrote:
>>>
>>> On 10/03/2013 02:44 PM, Abhay Sachan wrote:
>>>
>>> Hi All,
>>> I have tried setting up a ceph cluster with 3 nodes (3
>>> monitors). I am
>>> using RHEL 6.4 as OS with dumpling(0.67.3) release. Ceph cluster
>>> creation (using ceph-deploy as well as mkcephfs),
>>> ceph-creates-keys
>>> doesn't return on any of the servers. Whereas, if I create a
>>> cluster
>>> with only 1 node (1 monitor), key creation goes through. Has
>>> anybody
>>> seen this problem or any ideas what I might be missing??
>>>
>>> Regards,
>>> Abhay
>>>
>>>
>>> Those symptoms tell me that your monitors are not forming quorum.
>>> 'ceph-create-keys' needs the monitors to first establish a quorum,
>>> otherwise it will hang waiting for that to happen.
>>>
>>> Please make sure all your monitors are running. If so, try running
>>> 'ceph -s' on your cluster. If that hangs as well, try accessing
>>> each monitor's admin socket to check what's happening [1]. If that
>>> too fails, try looking into the logs for something obviously wrong.
>>> If you are not able to discern anything useful at that point,
>>> upload the logs to some place and point us to them -- we'll then be
>>> happy to take a look.
>>>
>>> Hope this helps.
>>>
>>> -Joao
>>>
>>> --
>>> Joao Eduardo Luis
>>> Software Engineer | http://inktank.com | http://ceph.com
>>>
>>>
>>>
>>
>> --
>> Joao Eduardo Luis
>> Software Engineer | http://inktank.com | http://ceph.com
>>
>
>
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
