In that case - yes...put everything on 1 card - or if both cards are 1G (or same speed for that matter...) - then you might want toblock all external traffic except i.e. SSH, WEB, but allow ALL traffic between all CEPH OSDs... so you can still use that network for "public/client" traffic - not sure how do you connect/use CEPH - from internet ??? or you have some more VMs/servers/clients on 192.* network... ?
On 14 March 2015 at 19:38, Georgios Dimitrakakis <[email protected]> wrote: > Andrija, > > I have two cards! > > One on 15.12.* and one on 192.* > > Obviously the 15.12.* is the external network (real public IP address e.g > used to access the node via SSH) > > That's why I am telling that my public network for CEPH is the 192. and > should I use the cluster network for that as well? > > Best, > > George > > > Georgios, >> >> no need to put ANYTHING if you dont plan to split client-to-OSD vs >> OSD-OSD-replication on 2 different Network Cards/Networks - for >> pefromance reasons. >> >> if you have only 1 network - simply DONT configure networks at all >> inside your CEPH.conf file... >> >> if you have 2 x 1G cards in servers, then you may use first 1G for >> client traffic, and second 1G for OSD-to-OSD replication... >> >> best >> >> On 14 March 2015 at 19:33, Georgios Dimitrakakis wrote: >> >> Andrija, >>> >>> Thanks for you help! >>> >>> In my case I just have one 192.* network, so should I put that for >>> both? >>> >>> Besides monitors do I have to list OSDs as well? >>> >>> Thanks again! >>> >>> Best, >>> >>> George >>> >>> This is how I did it, and then retart each OSD one by one, but >>>> monritor with ceph -s, when ceph is healthy, proceed with next >>>> OSD >>>> restart... >>>> Make sure the networks are fine on physical nodes, that you can >>>> ping >>>> in between... >>>> >>>> [global] >>>> x >>>> x >>>> x >>>> x >>>> x >>>> x >>>> >>>> ############################################# >>>> ### REPLICATION NETWORK ON SEPARATE 10G NICs >>>> >>>> # replication network >>>> cluster network = MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN >>>> MALICIOUS: MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN >>>> MALICIOUS: 10.44.251.0/24 [29] [29] >>>> >>>> # public/client network >>>> public network = MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN >>>> MALICIOUS: MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN >>>> MALICIOUS: 10.44.253.0/16 [30] [30] >>>> >>>> ############################################# >>>> >>>> [mon.xx] >>>> mon_addr = x.x.x.x:6789 >>>> host = xx >>>> >>>> [mon.yy] >>>> mon_addr = x.x.x.x:6789 >>>> host = yy >>>> >>>> [mon.zz] >>>> mon_addr = x.x.x.x:6789 >>>> host = zz >>>> >>>> On 14 March 2015 at 19:14, Georgios Dimitrakakis wrote: >>>> >>>> I thought that it was easy but apparently its not! >>>>> >>>>> I have the following in my conf file >>>>> >>>>> mon_host = 192.168.1.100,192.168.1.101,192.168.1.102 >>>>> public_network = MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN >>>>> MALICIOUS: MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN >>>>> MALICIOUS: 192.168.1.0/24 [26] [26] >>>>> mon_initial_members = fu,rai,jin >>>>> >>>>> but still the 15.12.6.21 link is being saturated.... >>>>> >>>>> Any ideas why??? >>>>> >>>>> Should I put cluster network as well?? >>>>> >>>>> Should I put each OSD in the CONF file??? >>>>> >>>>> Regards, >>>>> >>>>> George >>>>> >>>>> Andrija, >>>>>> >>>>>> thanks a lot for the useful info! >>>>>> >>>>>> I would also like to thank "Kingrat" at the IRC channel for >>>>>> his >>>>>> useful advice! >>>>>> >>>>>> I was under the wrong impression that public is the one used >>>>>> for >>>>>> RADOS. >>>>>> >>>>>> So I thought that public=external=internet and therefore I >>>>>> used >>>>>> that >>>>>> one in my conf. >>>>>> >>>>>> I understand now that I should have specified in CEPH Publics >>>>>> Network what I call >>>>>> "internal" and which is the one that all machines are talking >>>>>> directly to each other. >>>>>> >>>>>> Thanks you all for the feedback! >>>>>> >>>>>> Regards, >>>>>> >>>>>> George >>>>>> >>>>>> Public network is clients-to-OSD traffic - and if you have >>>>>>> NOT >>>>>>> explicitely defined cluster network, than also OSD-to-OSD >>>>>>> replication >>>>>>> takes place over same network. >>>>>>> >>>>>>> Otherwise, you can define public and cluster(private) >>>>>>> network - >>>>>>> so OSD >>>>>>> replication will happen over dedicated NICs (cluster >>>>>>> network) >>>>>>> and thus >>>>>>> speed up. >>>>>>> >>>>>>> If i.e. replica count on pool is 3, that means, each 1GB of >>>>>>> data >>>>>>> writen to some particualr OSD, will generate 3 x 1GB of >>>>>>> more >>>>>>> writes, >>>>>>> to the replicas... - which ideally will take place over >>>>>>> separate NICs >>>>>>> to speed up things... >>>>>>> >>>>>>> On 14 March 2015 at 17:43, Georgios Dimitrakakis wrote: >>>>>>> >>>>>>> Hi all!! >>>>>>>> >>>>>>>> What is the meaning of public_network in ceph.conf? >>>>>>>> >>>>>>>> Is it the network that OSDs are talking and transferring >>>>>>>> data? >>>>>>>> >>>>>>>> I have two nodes with two IP addresses each. One for >>>>>>>> internal >>>>>>>> network MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN >>>>>>>> MALICIOUS: >>>>>>>> MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>>>>>> MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>>>>>> 192.168.1.0/24 [1] [1] [1] >>>>>>>> and one external 15.12.6.* >>>>>>>> >>>>>>>> I see the following in my logs: >>>>>>>> >>>>>>>> osd.0 is down since epoch 2204, last address MAILSCANNER >>>>>>>> WARNING: >>>>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: MAILSCANNER WARNING: >>>>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: MAILSCANNER WARNING: >>>>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: 15.12.6.21:6826/33094 >>>>>>>> [2] [2] >>>>>>>> [2] >>>>>>>> osd.1 is down since epoch 2206, last address MAILSCANNER >>>>>>>> WARNING: >>>>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: MAILSCANNER WARNING: >>>>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: MAILSCANNER WARNING: >>>>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: 15.12.6.21:6817/32463 >>>>>>>> [3] [3] >>>>>>>> [3] >>>>>>>> osd.2 is down since epoch 2198, last address MAILSCANNER >>>>>>>> WARNING: >>>>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: MAILSCANNER WARNING: >>>>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: MAILSCANNER WARNING: >>>>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: 15.12.6.21:6843/34921 >>>>>>>> [4] [4] >>>>>>>> [4] >>>>>>>> osd.3 is down since epoch 2200, last address MAILSCANNER >>>>>>>> WARNING: >>>>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: MAILSCANNER WARNING: >>>>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: MAILSCANNER WARNING: >>>>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: 15.12.6.21:6838/34208 >>>>>>>> [5] [5] >>>>>>>> [5] >>>>>>>> osd.4 is down since epoch 2202, last address MAILSCANNER >>>>>>>> WARNING: >>>>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: MAILSCANNER WARNING: >>>>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: MAILSCANNER WARNING: >>>>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: 15.12.6.21:6831/33610 >>>>>>>> [6] [6] >>>>>>>> [6] >>>>>>>> osd.5 is down since epoch 2194, last address MAILSCANNER >>>>>>>> WARNING: >>>>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: MAILSCANNER WARNING: >>>>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: MAILSCANNER WARNING: >>>>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: 15.12.6.21:6858/35948 >>>>>>>> [7] [7] >>>>>>>> [7] >>>>>>>> osd.7 is down since epoch 2192, last address MAILSCANNER >>>>>>>> WARNING: >>>>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: MAILSCANNER WARNING: >>>>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: MAILSCANNER WARNING: >>>>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: 15.12.6.21:6871/36720 >>>>>>>> [8] [8] >>>>>>>> [8] >>>>>>>> osd.8 is down since epoch 2196, last address MAILSCANNER >>>>>>>> WARNING: >>>>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: MAILSCANNER WARNING: >>>>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: MAILSCANNER WARNING: >>>>>>>> NUMERICAL LINKS ARE OFTEN MALICIOUS: 15.12.6.21:6855/35354 >>>>>>>> [9] [9] >>>>>>>> [9] >>>>>>>> >>>>>>>> I ve managed to add a second node and during rebalancing >>>>>>>> I >>>>>>>> see that >>>>>>>> data is transfered through >>>>>>>> the internal 192.* but the external link is also >>>>>>>> saturated! >>>>>>>> >>>>>>>> What is being transferred from that? >>>>>>>> >>>>>>>> Any help much appreciated! >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> George >>>>>>>> _______________________________________________ >>>>>>>> ceph-users mailing list >>>>>>>> [email protected] [10] [10] [10] >>>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>>>>>> [11] [11] >>>>>>>> [11] >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Andrija Panić >>>>>>> >>>>>>> Links: >>>>>>> ------ >>>>>>> [1] MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN >>>>>>> MALICIOUS: >>>>>>> MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>>>>> http://192.168.1.0/24 [12] [12] >>>>>>> [2] MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN >>>>>>> MALICIOUS: >>>>>>> MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>>>>> http://15.12.6.21:6826/33094 [13] [13] >>>>>>> [3] MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN >>>>>>> MALICIOUS: >>>>>>> MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>>>>> http://15.12.6.21:6817/32463 [14] [14] >>>>>>> [4] MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN >>>>>>> MALICIOUS: >>>>>>> MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>>>>> http://15.12.6.21:6843/34921 [15] [15] >>>>>>> [5] MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN >>>>>>> MALICIOUS: >>>>>>> MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>>>>> http://15.12.6.21:6838/34208 [16] [16] >>>>>>> [6] MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN >>>>>>> MALICIOUS: >>>>>>> MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>>>>> http://15.12.6.21:6831/33610 [17] [17] >>>>>>> [7] MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN >>>>>>> MALICIOUS: >>>>>>> MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>>>>> http://15.12.6.21:6858/35948 [18] [18] >>>>>>> [8] MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN >>>>>>> MALICIOUS: >>>>>>> MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>>>>> http://15.12.6.21:6871/36720 [19] [19] >>>>>>> [9] MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN >>>>>>> MALICIOUS: >>>>>>> MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>>>>> http://15.12.6.21:6855/35354 [20] [20] >>>>>>> [10] mailto:[email protected] [21] [21] >>>>>>> [11] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>>>>> [22] >>>>>>> [22] >>>>>>> [12] mailto:[email protected] [23] [23] >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> ceph-users mailing list >>>>>> [email protected] [24] [24] >>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com [25] >>>>>> [25] >>>>>> >>>>> >>>>> _______________________________________________ >>>>> ceph-users mailing list >>>>> [email protected] [27] [27] >>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com [28] >>>>> [28] >>>>> >>>> >>>> -- >>>> >>>> Andrija Panić >>>> >>>> Links: >>>> ------ >>>> [1] MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>> http://192.168.1.0/24 [31] >>>> [2] MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>> http://15.12.6.21:6826/33094 [32] >>>> [3] MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>> http://15.12.6.21:6817/32463 [33] >>>> [4] MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>> http://15.12.6.21:6843/34921 [34] >>>> [5] MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>> http://15.12.6.21:6838/34208 [35] >>>> [6] MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>> http://15.12.6.21:6831/33610 [36] >>>> [7] MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>> http://15.12.6.21:6858/35948 [37] >>>> [8] MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>> http://15.12.6.21:6871/36720 [38] >>>> [9] MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>> http://15.12.6.21:6855/35354 [39] >>>> [10] mailto:[email protected] [40] >>>> [11] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com [41] >>>> [12] MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>> http://192.168.1.0/24 [42] >>>> [13] MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>> http://15.12.6.21:6826/33094 [43] >>>> [14] MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>> http://15.12.6.21:6817/32463 [44] >>>> [15] MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>> http://15.12.6.21:6843/34921 [45] >>>> [16] MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>> http://15.12.6.21:6838/34208 [46] >>>> [17] MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>> http://15.12.6.21:6831/33610 [47] >>>> [18] MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>> http://15.12.6.21:6858/35948 [48] >>>> [19] MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>> http://15.12.6.21:6871/36720 [49] >>>> [20] MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>> http://15.12.6.21:6855/35354 [50] >>>> [21] mailto:[email protected] [51] >>>> [22] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com [52] >>>> [23] mailto:[email protected] [53] >>>> [24] mailto:[email protected] [54] >>>> [25] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com [55] >>>> [26] MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>> http://192.168.1.0/24 [56] >>>> [27] mailto:[email protected] [57] >>>> [28] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com [58] >>>> [29] MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>> http://10.44.251.0/24 [59] >>>> [30] MAILSCANNER WARNING: NUMERICAL LINKS ARE OFTEN MALICIOUS: >>>> http://10.44.253.0/16 [60] >>>> [31] mailto:[email protected] [61] >>>> >>> >> -- >> >> Andrija Panić >> >> Links: >> ------ >> [1] http://192.168.1.0/24 >> [2] http://15.12.6.21:6826/33094 >> [3] http://15.12.6.21:6817/32463 >> [4] http://15.12.6.21:6843/34921 >> [5] http://15.12.6.21:6838/34208 >> [6] http://15.12.6.21:6831/33610 >> [7] http://15.12.6.21:6858/35948 >> [8] http://15.12.6.21:6871/36720 >> [9] http://15.12.6.21:6855/35354 >> [10] mailto:[email protected] >> [11] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> [12] http://192.168.1.0/24 >> [13] http://15.12.6.21:6826/33094 >> [14] http://15.12.6.21:6817/32463 >> [15] http://15.12.6.21:6843/34921 >> [16] http://15.12.6.21:6838/34208 >> [17] http://15.12.6.21:6831/33610 >> [18] http://15.12.6.21:6858/35948 >> [19] http://15.12.6.21:6871/36720 >> [20] http://15.12.6.21:6855/35354 >> [21] mailto:[email protected] >> [22] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> [23] mailto:[email protected] >> [24] mailto:[email protected] >> [25] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> [26] http://192.168.1.0/24 >> [27] mailto:[email protected] >> [28] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> [29] http://10.44.251.0/24 >> [30] http://10.44.253.0/16 >> [31] http://192.168.1.0/24 >> [32] http://15.12.6.21:6826/33094 >> [33] http://15.12.6.21:6817/32463 >> [34] http://15.12.6.21:6843/34921 >> [35] http://15.12.6.21:6838/34208 >> [36] http://15.12.6.21:6831/33610 >> [37] http://15.12.6.21:6858/35948 >> [38] http://15.12.6.21:6871/36720 >> [39] http://15.12.6.21:6855/35354 >> [40] mailto:[email protected] >> [41] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> [42] http://192.168.1.0/24 >> [43] http://15.12.6.21:6826/33094 >> [44] http://15.12.6.21:6817/32463 >> [45] http://15.12.6.21:6843/34921 >> [46] http://15.12.6.21:6838/34208 >> [47] http://15.12.6.21:6831/33610 >> [48] http://15.12.6.21:6858/35948 >> [49] http://15.12.6.21:6871/36720 >> [50] http://15.12.6.21:6855/35354 >> [51] mailto:[email protected] >> [52] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> [53] mailto:[email protected] >> [54] mailto:[email protected] >> [55] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> [56] http://192.168.1.0/24 >> [57] mailto:[email protected] >> [58] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> [59] http://10.44.251.0/24 >> [60] http://10.44.253.0/16 >> [61] mailto:[email protected] >> [62] mailto:[email protected] >> > > -- > -- Andrija Panić
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
