Georgios, no need to put ANYTHING if you don't plan to split client-to-OSD vs OSD-OSD-replication on 2 different Network Cards/Networks - for pefromance reasons.
if you have only 1 network - simply DONT configure networks at all inside your CEPH.conf file... if you have 2 x 1G cards in servers, then you may use first 1G for client traffic, and second 1G for OSD-to-OSD replication... best On 14 March 2015 at 19:33, Georgios Dimitrakakis <[email protected]> wrote: > Andrija, > > Thanks for you help! > > In my case I just have one 192.* network, so should I put that for both? > > Besides monitors do I have to list OSDs as well? > > Thanks again! > > Best, > > George > > This is how I did it, and then retart each OSD one by one, but >> monritor with ceph -s, when ceph is healthy, proceed with next OSD >> restart... >> Make sure the networks are fine on physical nodes, that you can ping >> in between... >> >> [global] >> x >> x >> x >> x >> x >> x >> >> ############################################# >> ### REPLICATION NETWORK ON SEPARATE 10G NICs >> >> # replication network >> cluster network = MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN >> MALICIOUS: 10.44.251.0/24 [29] >> >> # public/client network >> public network = MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN >> MALICIOUS: 10.44.253.0/16 [30] >> >> ############################################# >> >> [mon.xx] >> mon_addr = x.x.x.x:6789 >> host = xx >> >> [mon.yy] >> mon_addr = x.x.x.x:6789 >> host = yy >> >> [mon.zz] >> mon_addr = x.x.x.x:6789 >> host = zz >> >> On 14 March 2015 at 19:14, Georgios Dimitrakakis wrote: >> >> I thought that it was easy but apparently its not! >>> >>> I have the following in my conf file >>> >>> mon_host = 192.168.1.100,192.168.1.101,192.168.1.102 >>> public_network = MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN >>> MALICIOUS: 192.168.1.0/24 [26] >>> mon_initial_members = fu,rai,jin >>> >>> but still the 15.12.6.21 link is being saturated.... >>> >>> Any ideas why??? >>> >>> Should I put cluster network as well?? >>> >>> Should I put each OSD in the CONF file??? >>> >>> Regards, >>> >>> George >>> >>> Andrija, >>>> >>>> thanks a lot for the useful info! >>>> >>>> I would also like to thank "Kingrat" at the IRC channel for his >>>> useful advice! >>>> >>>> I was under the wrong impression that public is the one used for >>>> RADOS. >>>> >>>> So I thought that public=external=internet and therefore I used >>>> that >>>> one in my conf. >>>> >>>> I understand now that I should have specified in CEPH Publics >>>> Network what I call >>>> "internal" and which is the one that all machines are talking >>>> directly to each other. >>>> >>>> Thanks you all for the feedback! >>>> >>>> Regards, >>>> >>>> George >>>> >>>> Public network is clients-to-OSD traffic - and if you have NOT >>>>> explicitely defined cluster network, than also OSD-to-OSD >>>>> replication >>>>> takes place over same network. >>>>> >>>>> Otherwise, you can define public and cluster(private) network - >>>>> so OSD >>>>> replication will happen over dedicated NICs (cluster network) >>>>> and thus >>>>> speed up. >>>>> >>>>> If i.e. replica count on pool is 3, that means, each 1GB of >>>>> data >>>>> writen to some particualr OSD, will generate 3 x 1GB of more >>>>> writes, >>>>> to the replicas... - which ideally will take place over >>>>> separate NICs >>>>> to speed up things... >>>>> >>>>> On 14 March 2015 at 17:43, Georgios Dimitrakakis wrote: >>>>> >>>>> Hi all!! >>>>>> >>>>>> What is the meaning of public_network in ceph.conf? >>>>>> >>>>>> Is it the network that OSDs are talking and transferring >>>>>> data? >>>>>> >>>>>> I have two nodes with two IP addresses each. One for internal >>>>>> network MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN >>>>>> MALICIOUS: >>>>>> MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>>>> 192.168.1.0/24 [1] [1] >>>>>> and one external 15.12.6.* >>>>>> >>>>>> I see the following in my logs: >>>>>> >>>>>> osd.0 is down since epoch 2204, last address MAILSCANNER >>>>>> WARNING: >>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: MAILSCANNER WARNING: >>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: 15.12.6.21:6826/33094 [2] >>>>>> [2] >>>>>> osd.1 is down since epoch 2206, last address MAILSCANNER >>>>>> WARNING: >>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: MAILSCANNER WARNING: >>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: 15.12.6.21:6817/32463 [3] >>>>>> [3] >>>>>> osd.2 is down since epoch 2198, last address MAILSCANNER >>>>>> WARNING: >>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: MAILSCANNER WARNING: >>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: 15.12.6.21:6843/34921 [4] >>>>>> [4] >>>>>> osd.3 is down since epoch 2200, last address MAILSCANNER >>>>>> WARNING: >>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: MAILSCANNER WARNING: >>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: 15.12.6.21:6838/34208 [5] >>>>>> [5] >>>>>> osd.4 is down since epoch 2202, last address MAILSCANNER >>>>>> WARNING: >>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: MAILSCANNER WARNING: >>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: 15.12.6.21:6831/33610 [6] >>>>>> [6] >>>>>> osd.5 is down since epoch 2194, last address MAILSCANNER >>>>>> WARNING: >>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: MAILSCANNER WARNING: >>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: 15.12.6.21:6858/35948 [7] >>>>>> [7] >>>>>> osd.7 is down since epoch 2192, last address MAILSCANNER >>>>>> WARNING: >>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: MAILSCANNER WARNING: >>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: 15.12.6.21:6871/36720 [8] >>>>>> [8] >>>>>> osd.8 is down since epoch 2196, last address MAILSCANNER >>>>>> WARNING: >>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: MAILSCANNER WARNING: >>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: 15.12.6.21:6855/35354 [9] >>>>>> [9] >>>>>> >>>>>> I ve managed to add a second node and during rebalancing I >>>>>> see that >>>>>> data is transfered through >>>>>> the internal 192.* but the external link is also saturated! >>>>>> >>>>>> What is being transferred from that? >>>>>> >>>>>> Any help much appreciated! >>>>>> >>>>>> Regards, >>>>>> >>>>>> George >>>>>> _______________________________________________ >>>>>> ceph-users mailing list >>>>>> [email protected] [10] [10] >>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com [11] >>>>>> [11] >>>>>> >>>>> >>>>> -- >>>>> >>>>> Andrija Panić >>>>> >>>>> Links: >>>>> ------ >>>>> [1] MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>>> http://192.168.1.0/24 [12] >>>>> [2] MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>>> http://15.12.6.21:6826/33094 [13] >>>>> [3] MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>>> http://15.12.6.21:6817/32463 [14] >>>>> [4] MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>>> http://15.12.6.21:6843/34921 [15] >>>>> [5] MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>>> http://15.12.6.21:6838/34208 [16] >>>>> [6] MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>>> http://15.12.6.21:6831/33610 [17] >>>>> [7] MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>>> http://15.12.6.21:6858/35948 [18] >>>>> [8] MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>>> http://15.12.6.21:6871/36720 [19] >>>>> [9] MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>>> http://15.12.6.21:6855/35354 [20] >>>>> [10] mailto:[email protected] [21] >>>>> [11] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>>> [22] >>>>> [12] mailto:[email protected] [23] >>>>> >>>> >>>> _______________________________________________ >>>> ceph-users mailing list >>>> [email protected] [24] >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com [25] >>>> >>> >>> _______________________________________________ >>> ceph-users mailing list >>> [email protected] [27] >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com [28] >>> >> >> -- >> >> Andrija Panić >> >> Links: >> ------ >> [1] http://192.168.1.0/24 >> [2] http://15.12.6.21:6826/33094 >> [3] http://15.12.6.21:6817/32463 >> [4] http://15.12.6.21:6843/34921 >> [5] http://15.12.6.21:6838/34208 >> [6] http://15.12.6.21:6831/33610 >> [7] http://15.12.6.21:6858/35948 >> [8] http://15.12.6.21:6871/36720 >> [9] http://15.12.6.21:6855/35354 >> [10] mailto:[email protected] >> [11] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> [12] http://192.168.1.0/24 >> [13] http://15.12.6.21:6826/33094 >> [14] http://15.12.6.21:6817/32463 >> [15] http://15.12.6.21:6843/34921 >> [16] http://15.12.6.21:6838/34208 >> [17] http://15.12.6.21:6831/33610 >> [18] http://15.12.6.21:6858/35948 >> [19] http://15.12.6.21:6871/36720 >> [20] http://15.12.6.21:6855/35354 >> [21] mailto:[email protected] >> [22] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> [23] mailto:[email protected] >> [24] mailto:[email protected] >> [25] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> [26] http://192.168.1.0/24 >> [27] mailto:[email protected] >> [28] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> [29] http://10.44.251.0/24 >> [30] http://10.44.253.0/16 >> [31] mailto:[email protected] >> > > -- Andrija Panić
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
