Re: [ceph-users] ceph-deploy: is it a requirement that the name of each node of the ceph cluster must be resolved to the public IP ?

[Massimo Sgaravatto]

This [*] is my ceph.conf

10.70.42.9 is the public address

And it is indeed the IP used by the MON daemon:

[root@c-mon-02 ~]# netstat -anp | grep 6789
tcp0  0 10.70.42.9:6789 0.0.0.0:*   LISTEN
3835/ceph-mon
tcp0  0 10.70.42.9:3359210.70.42.10:6789
ESTABLISHED 3835/ceph-mon
tcp0  0 10.70.42.9:4178610.70.42.8:6789
 ESTABLISHED 3835/ceph-mon
tcp   106008  0 10.70.42.9:3321010.70.42.10:6789
CLOSE_WAIT  1162/ceph-mgr
tcp   100370  0 10.70.42.9:3321810.70.42.10:6789
CLOSE_WAIT  1162/ceph-mgr
tcp0  0 10.70.42.9:3357810.70.42.10:6789
ESTABLISHED 1162/ceph-mgr


But the command:

/usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon.
--keyring=/var/lib/ceph/mon/ceph-c-mon-02/keyring auth get client.admin
exported keyring for client.admin

fails with c-mon-02 resolved to the management IP.

As wokaround I can add in /etc/hosts the mapping with the public address:

10.70.42.9  c-mon-02


but I wonder if this is the expected behavior


Cheers, Massimo

[*]

[global]
fsid = 7a8cb8ff-562b-47da-a6aa-507136587dcf
public network = 10.70.42.0/24
cluster network = 10.69.42.0/24


auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

osd pool default size = 3  # Write an object 3 times.
osd pool default min size = 2


osd pool default pg num = 128
osd pool default pgp num = 128


[mon]
mon host =  c-mon-01, c-mon-02, c-mon-03
mon addr =  10.70.42.10:6789, 10.70.42.9:6789, 10.70.42.8:6789

[mon.c-mon-01]
mon addr = 10.70.42.10:6789
host = c-mon-01

[mon.c-mon-02]
mon addr = 10.70.42.9:6789
host = c-mon-02

[mon.c-mon-03]
mon addr = 10.70.42.8:6789
host = c-mon-03

[osd]
osd mount options xfs = rw,noatime,inode64,logbufs=8,logbsize=256k



On Thu, May 10, 2018 at 1:12 PM, Paul Emmerich 
wrote:

> check ceph.conf, it controls to which mon IP the client tries to connect.
>
> 2018-05-10 12:57 GMT+02:00 Massimo Sgaravatto <
> massimo.sgarava...@gmail.com>:
>
>> I configured the "public network" attribute in the ceph configuration
>> file.
>>
>> But it looks like to me that in the "auth get client.admin" command [*]
>> issued by ceph-deploy the address of the management network is used (I
>> guess because c-mon-02 gets resolved to the IP management address)
>>
>> Cheers, Massimo
>>
>> [*]
>> /usr/bin/ceph  --connect-timeout=25 --cluster=ceph --name mon.
>> --keyring=/var/lib/ceph/mon/ceph-c-mon-02/keyring auth get client.admin
>>
>> On Thu, May 10, 2018 at 12:49 PM, Paul Emmerich 
>> wrote:
>>
>>> Monitors can use only exactly one IP address. ceph-deploy uses some
>>> heuristics
>>> based on hostname resolution and ceph public addr configuration to guess
>>> which
>>> one to use during setup. (Which I've always found to be a quite annoying
>>> feature.)
>>>
>>> The mon's IP must be reachable from all ceph daemons and clients, so it
>>> should be
>>> on your "public" network. Changing the IP of a mon is possible but
>>> annoying, it is
>>> often easier to remove and then re-add with a new IP (if possible):
>>>
>>> http://docs.ceph.com/docs/master/rados/operations/add-or-rm-
>>> mons/#changing-a-monitor-s-ip-address
>>>
>>>
>>> Paul
>>>
>>> 2018-05-10 12:36 GMT+02:00 Massimo Sgaravatto <
>>> massimo.sgarava...@gmail.com>:
>>>
 I have a ceph cluster that I manually deployed, and now I am trying to
 see if I can use ceph-deploy to deploy new nodes (in particular the object
 gw).

 The network configuration is the following:

 Each MON node has two network IP: one on a "management network" (not
 used for ceph related stuff) and one on a "public network",
 The MON daemon listens to on the pub network

 Each OSD node  has three network IPs: one on a "management network"
 (not used for ceph related stuff), one on a "public network" and the third
 one is an internal network to be used as ceph cluster network (for ceph
 internal traffic: replication, recovery, etc)


 Name resolution works, but names are resolved to the IP address of the
 management network.
 And it looks like this is a problem. E.g. the following command (used
 in ceph-deploy gatherkeys) issued on a MON host (c-mon-02) doesn't work:

 /usr/bin/ceph --verbose --connect-timeout=25 --cluster=ceph --name mon.
 --keyring=/var/lib/ceph/mon/ceph-c-mon-02/keyring auth get client.admin

 unless I change the name resolution of c-mon-02 to the public address


 Is it a requirement (at least for ceph-deploy) that the name of each
 node of the ceph cluster must be resolved to the public IP address ?


 Thanks, Massimo

 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


>>>
>>>
>>> --
>>> --
>>> Paul Emmerich
>>>
>>> 

Re: [ceph-users] ceph-deploy: is it a requirement that the name of each node of the ceph cluster must be resolved to the public IP ?

[Paul Emmerich]

check ceph.conf, it controls to which mon IP the client tries to connect.

2018-05-10 12:57 GMT+02:00 Massimo Sgaravatto 
:

> I configured the "public network" attribute in the ceph configuration file.
>
> But it looks like to me that in the "auth get client.admin" command [*]
> issued by ceph-deploy the address of the management network is used (I
> guess because c-mon-02 gets resolved to the IP management address)
>
> Cheers, Massimo
>
> [*]
> /usr/bin/ceph  --connect-timeout=25 --cluster=ceph --name mon.
> --keyring=/var/lib/ceph/mon/ceph-c-mon-02/keyring auth get client.admin
>
> On Thu, May 10, 2018 at 12:49 PM, Paul Emmerich 
> wrote:
>
>> Monitors can use only exactly one IP address. ceph-deploy uses some
>> heuristics
>> based on hostname resolution and ceph public addr configuration to guess
>> which
>> one to use during setup. (Which I've always found to be a quite annoying
>> feature.)
>>
>> The mon's IP must be reachable from all ceph daemons and clients, so it
>> should be
>> on your "public" network. Changing the IP of a mon is possible but
>> annoying, it is
>> often easier to remove and then re-add with a new IP (if possible):
>>
>> http://docs.ceph.com/docs/master/rados/operations/add-or-rm-
>> mons/#changing-a-monitor-s-ip-address
>>
>>
>> Paul
>>
>> 2018-05-10 12:36 GMT+02:00 Massimo Sgaravatto <
>> massimo.sgarava...@gmail.com>:
>>
>>> I have a ceph cluster that I manually deployed, and now I am trying to
>>> see if I can use ceph-deploy to deploy new nodes (in particular the object
>>> gw).
>>>
>>> The network configuration is the following:
>>>
>>> Each MON node has two network IP: one on a "management network" (not
>>> used for ceph related stuff) and one on a "public network",
>>> The MON daemon listens to on the pub network
>>>
>>> Each OSD node  has three network IPs: one on a "management network" (not
>>> used for ceph related stuff), one on a "public network" and the third one
>>> is an internal network to be used as ceph cluster network (for ceph
>>> internal traffic: replication, recovery, etc)
>>>
>>>
>>> Name resolution works, but names are resolved to the IP address of the
>>> management network.
>>> And it looks like this is a problem. E.g. the following command (used in
>>> ceph-deploy gatherkeys) issued on a MON host (c-mon-02) doesn't work:
>>>
>>> /usr/bin/ceph --verbose --connect-timeout=25 --cluster=ceph --name mon.
>>> --keyring=/var/lib/ceph/mon/ceph-c-mon-02/keyring auth get client.admin
>>>
>>> unless I change the name resolution of c-mon-02 to the public address
>>>
>>>
>>> Is it a requirement (at least for ceph-deploy) that the name of each
>>> node of the ceph cluster must be resolved to the public IP address ?
>>>
>>>
>>> Thanks, Massimo
>>>
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>>
>>
>>
>> --
>> --
>> Paul Emmerich
>>
>> Looking for help with your Ceph cluster? Contact us at https://croit.io
>>
>> croit GmbH
>> Freseniusstr. 31h
>> 
>> 81247 München
>> 
>> www.croit.io
>> Tel: +49 89 1896585 90
>>
>
>


-- 
-- 
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-deploy: is it a requirement that the name of each node of the ceph cluster must be resolved to the public IP ?

[Massimo Sgaravatto]

I configured the "public network" attribute in the ceph configuration file.

But it looks like to me that in the "auth get client.admin" command [*]
issued by ceph-deploy the address of the management network is used (I
guess because c-mon-02 gets resolved to the IP management address)

Cheers, Massimo

[*]
/usr/bin/ceph  --connect-timeout=25 --cluster=ceph --name mon.
--keyring=/var/lib/ceph/mon/ceph-c-mon-02/keyring auth get client.admin

On Thu, May 10, 2018 at 12:49 PM, Paul Emmerich 
wrote:

> Monitors can use only exactly one IP address. ceph-deploy uses some
> heuristics
> based on hostname resolution and ceph public addr configuration to guess
> which
> one to use during setup. (Which I've always found to be a quite annoying
> feature.)
>
> The mon's IP must be reachable from all ceph daemons and clients, so it
> should be
> on your "public" network. Changing the IP of a mon is possible but
> annoying, it is
> often easier to remove and then re-add with a new IP (if possible):
>
> http://docs.ceph.com/docs/master/rados/operations/add-
> or-rm-mons/#changing-a-monitor-s-ip-address
>
>
> Paul
>
> 2018-05-10 12:36 GMT+02:00 Massimo Sgaravatto <
> massimo.sgarava...@gmail.com>:
>
>> I have a ceph cluster that I manually deployed, and now I am trying to
>> see if I can use ceph-deploy to deploy new nodes (in particular the object
>> gw).
>>
>> The network configuration is the following:
>>
>> Each MON node has two network IP: one on a "management network" (not used
>> for ceph related stuff) and one on a "public network",
>> The MON daemon listens to on the pub network
>>
>> Each OSD node  has three network IPs: one on a "management network" (not
>> used for ceph related stuff), one on a "public network" and the third one
>> is an internal network to be used as ceph cluster network (for ceph
>> internal traffic: replication, recovery, etc)
>>
>>
>> Name resolution works, but names are resolved to the IP address of the
>> management network.
>> And it looks like this is a problem. E.g. the following command (used in
>> ceph-deploy gatherkeys) issued on a MON host (c-mon-02) doesn't work:
>>
>> /usr/bin/ceph --verbose --connect-timeout=25 --cluster=ceph --name mon.
>> --keyring=/var/lib/ceph/mon/ceph-c-mon-02/keyring auth get client.admin
>>
>> unless I change the name resolution of c-mon-02 to the public address
>>
>>
>> Is it a requirement (at least for ceph-deploy) that the name of each node
>> of the ceph cluster must be resolved to the public IP address ?
>>
>>
>> Thanks, Massimo
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>
>
> --
> --
> Paul Emmerich
>
> Looking for help with your Ceph cluster? Contact us at https://croit.io
>
> croit GmbH
> Freseniusstr. 31h
> 
> 81247 München
> 
> www.croit.io
> Tel: +49 89 1896585 90
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-deploy: is it a requirement that the name of each node of the ceph cluster must be resolved to the public IP ?

[Paul Emmerich]

Monitors can use only exactly one IP address. ceph-deploy uses some
heuristics
based on hostname resolution and ceph public addr configuration to guess
which
one to use during setup. (Which I've always found to be a quite annoying
feature.)

The mon's IP must be reachable from all ceph daemons and clients, so it
should be
on your "public" network. Changing the IP of a mon is possible but
annoying, it is
often easier to remove and then re-add with a new IP (if possible):

http://docs.ceph.com/docs/master/rados/operations/add-or-rm-mons/#changing-a-monitor-s-ip-address


Paul

2018-05-10 12:36 GMT+02:00 Massimo Sgaravatto 
:

> I have a ceph cluster that I manually deployed, and now I am trying to see
> if I can use ceph-deploy to deploy new nodes (in particular the object gw).
>
> The network configuration is the following:
>
> Each MON node has two network IP: one on a "management network" (not used
> for ceph related stuff) and one on a "public network",
> The MON daemon listens to on the pub network
>
> Each OSD node  has three network IPs: one on a "management network" (not
> used for ceph related stuff), one on a "public network" and the third one
> is an internal network to be used as ceph cluster network (for ceph
> internal traffic: replication, recovery, etc)
>
>
> Name resolution works, but names are resolved to the IP address of the
> management network.
> And it looks like this is a problem. E.g. the following command (used in
> ceph-deploy gatherkeys) issued on a MON host (c-mon-02) doesn't work:
>
> /usr/bin/ceph --verbose --connect-timeout=25 --cluster=ceph --name mon.
> --keyring=/var/lib/ceph/mon/ceph-c-mon-02/keyring auth get client.admin
>
> unless I change the name resolution of c-mon-02 to the public address
>
>
> Is it a requirement (at least for ceph-deploy) that the name of each node
> of the ceph cluster must be resolved to the public IP address ?
>
>
> Thanks, Massimo
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>


-- 
-- 
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com