I don't know. Ceph documentation on Mimic doesn't appear to go into too much details on RDMA in general, but still it's mentioned in the Ceph docs here and there. Some examples: Change log - http://docs.ceph.com/docs/master/releases/mimic/ <http://docs.ceph.com/docs/master/releases/mimic/> Async messenger options - http://docs.ceph.com/docs/master/rados/configuration/ms-ref/ <http://docs.ceph.com/docs/master/rados/configuration/ms-ref/>
I want to believe that the official docs wouldn't mention something that's completely broken? There are multiple posts in this very mailing list from people trying to make it work. -- Michael Green Customer Support & Integration Tel. +1 (518) 9862385 [email protected] E8 Storage has a new look, find out more <https://e8storage.com/when-performance-matters-a-new-look-for-e8-storage/> > On Dec 18, 2018, at 6:55 AM, Виталий Филиппов <[email protected]> wrote: > > Is RDMA officially supported? I'm asking because I recently tried to use DPDK > and it seems it's broken... i.e the code is there, but does not compile until > I fix cmake scripts, and after fixing the build OSDs just get segfaults and > die after processing something like 40-50 incoming packets. > > Maybe RDMA is in the same state? > > 13 декабря 2018 г. 2:42:23 GMT+03:00, Michael Green <[email protected]> > пишет: > Sorry for bumping the thread. I refuse to believe there are no people on this > list who have successfully enabled and run RDMA with Mimic. :) > > Mike > >> Hello collective wisdom, >> >> ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) mimic >> (stable) here. >> >> I have a working cluster here consisting of 3 monitor hosts, 64 OSD >> processes across 4 osd hosts, plus 2 MDSs, plus 2 MGRs. All of that is >> consumed by 10 client nodes. >> >> Every host in the cluster, including clients is >> RHEL 7.5 >> Mellanox OFED 4.4-2.0.7.0 >> RoCE NICs are either MCX416A-CCAT or MCX414A-CCAT @ 50Gbit/sec >> The NICs are all mlx5_0 port 1 >> >> ring and ib_send_bw work fine both ways on any two nodes in the cluster. >> >> Full configuration of the cluster is pasted below, but RDMA related >> parameters are configured as following: >> >> >> ms_public_type = async+rdma >> ms_cluster = async+rdma >> # Exclude clients for now >> ms_type = async+posix >> >> ms_async_rdma_device_name = mlx5_0 >> ms_async_rdma_polling_us = 0 >> ms_async_rdma_port_num=1 >> >> When I try to start MON, it immediately fails as below. Anybody has seen >> this or could give any pointers what to/where to look next? >> >> >> ------ceph-mon.rio.log--begin------ >> 2018-12-12 22:35:30.011 7f515dc39140 0 set uid:gid to 167:167 (ceph:ceph) >> 2018-12-12 22:35:30.011 7f515dc39140 0 ceph version 13.2.2 >> (02899bfda814146b021136e9d8e80eba494e1126) mimic (stable), process ceph-mon, >> pid 2129843 >> 2018-12-12 22:35:30.011 7f515dc39140 0 pidfile_write: ignore empty >> --pid-file >> 2018-12-12 22:35:30.036 7f515dc39140 0 load: jerasure load: lrc load: isa >> 2018-12-12 22:35:30.036 7f515dc39140 0 set rocksdb option compression = >> kNoCompression >> 2018-12-12 22:35:30.036 7f515dc39140 0 set rocksdb option >> level_compaction_dynamic_level_bytes = true >> 2018-12-12 22:35:30.036 7f515dc39140 0 set rocksdb option >> write_buffer_size = 33554432 >> 2018-12-12 22:35:30.036 7f515dc39140 0 set rocksdb option compression = >> kNoCompression >> 2018-12-12 22:35:30.036 7f515dc39140 0 set rocksdb option >> level_compaction_dynamic_level_bytes = true >> 2018-12-12 22:35:30.036 7f515dc39140 0 set rocksdb option >> write_buffer_size = 33554432 >> 2018-12-12 22:35:30.147 7f51442ed700 2 Event(0x55d927e95700 nevent=5000 >> time_id=1).set_owner idx=1 owner=139987012998912 >> 2018-12-12 22:35:30.147 7f51442ed700 10 stack operator() starting >> 2018-12-12 22:35:30.147 7f5143aec700 2 Event(0x55d927e95200 nevent=5000 >> time_id=1).set_owner idx=0 owner=139987004606208 >> 2018-12-12 22:35:30.147 7f5144aee700 2 Event(0x55d927e95c00 nevent=5000 >> time_id=1).set_owner idx=2 owner=139987021391616 >> 2018-12-12 22:35:30.147 7f5143aec700 10 stack operator() starting >> 2018-12-12 22:35:30.147 7f5144aee700 10 stack operator() starting >> 2018-12-12 22:35:30.147 7f515dc39140 0 starting mon.rio rank 0 at public >> addr 192.168.1.58:6789/0 at bind addr 192.168.1.58:6789/0 mon_data >> /var/lib/ceph/mon/ceph-rio fsid 376540c8-a362-41cc-9a58-9c8ceca0e4ee >> 2018-12-12 22:35:30.147 7f515dc39140 10 -- - bind bind 192.168.1.58:6789/0 >> 2018-12-12 22:35:30.147 7f515dc39140 10 -- - bind Network Stack is not ready >> for bind yet - postponed >> 2018-12-12 22:35:30.147 7f515dc39140 0 starting mon.rio rank 0 at >> 192.168.1.58:6789/0 mon_data /var/lib/ceph/mon/ceph-rio fsid >> 376540c8-a362-41cc-9a58-9c8ceca0e4ee >> 2018-12-12 22:35:30.148 7f515dc39140 0 mon.rio@-1(probing).mds e84 new map >> 2018-12-12 22:35:30.148 7f515dc39140 0 mon.rio@-1(probing).mds e84 print_map >> e84 >> enable_multiple, ever_enabled_multiple: 0,0 >> compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable >> ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds >> uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file >> layout v2,10=snaprealm v2} >> legacy client fscid: -1 >> >> No filesystems configured >> Standby daemons: >> >> 5906437: 192.168.1.152:6800/1077205146 'prince' mds.-1.0 up:standby >> seq 2 >> 6284118: 192.168.1.59:6800/1266235911 'salvador' mds.-1.0 up:standby >> seq 2 >> >> 2018-12-12 22:35:30.148 7f515dc39140 0 mon.rio@-1(probing).osd e25894 crush >> map has features 288514051259236352, adjusting msgr requires >> 2018-12-12 22:35:30.148 7f515dc39140 0 mon.rio@-1(probing).osd e25894 crush >> map has features 288514051259236352, adjusting msgr requires >> 2018-12-12 22:35:30.148 7f515dc39140 0 mon.rio@-1(probing).osd e25894 crush >> map has features 1009089991638532096, adjusting msgr requires >> 2018-12-12 22:35:30.148 7f515dc39140 0 mon.rio@-1(probing).osd e25894 crush >> map has features 288514051259236352, adjusting msgr requires >> 2018-12-12 22:35:30.149 7f515dc39140 10 -- - create_connect >> 192.168.1.88:6800/1638, creating connection and registering >> 2018-12-12 22:35:30.149 7f515dc39140 10 -- - >> 192.168.1.88:6800/1638 >> conn(0x55d9281fbe00 :-1 s=STATE_NONE pgs=0 cs=0 l=0)._connect csq=0 >> 2018-12-12 22:35:30.149 7f515dc39140 10 -- - get_connection mgr.5894115 >> 192.168.1.88:6800/1638 new 0x55d9281fbe00 >> 2018-12-12 22:35:30.150 7f515dc39140 1 -- - --> 192.168.1.88:6800/1638 -- >> mgropen(unknown.rio) v3 -- 0x55d92844e000 con 0 >> 2018-12-12 22:35:30.151 7f515dc39140 1 -- - start start >> 2018-12-12 22:35:30.151 7f515dc39140 1 -- - start start >> 2018-12-12 22:35:30.151 7f515dc39140 10 -- - ready - >> 2018-12-12 22:35:30.151 7f515dc39140 10 -- - bind bind 192.168.1.58:6789/0 >> 2018-12-12 22:35:30.151 7f515dc39140 10 Processor -- bind >> 2018-12-12 22:35:30.154 7f5144aee700 1 Infiniband binding_port found active >> port 1 >> 2018-12-12 22:35:30.154 7f5144aee700 1 Infiniband init receive queue length >> is 4096 receive buffers >> 2018-12-12 22:35:30.154 7f5144aee700 1 Infiniband init assigning: 1024 send >> buffers >> 2018-12-12 22:35:30.154 7f5144aee700 1 Infiniband init device allow 4194303 >> completion entries >> 2018-12-12 22:35:30.509 7f515dc39140 10 Processor -- bind bound to >> 192.168.1.58:6789/0 >> 2018-12-12 22:35:30.509 7f515dc39140 1 -- 192.168.1.58:6789/0 learned_addr >> learned my addr 192.168.1.58:6789/0 >> 2018-12-12 22:35:30.509 7f515dc39140 1 -- 192.168.1.58:6789/0 _finish_bind >> bind my_inst.addr is 192.168.1.58:6789/0 >> 2018-12-12 22:35:30.510 7f515dc39140 10 -- - ready - >> 2018-12-12 22:35:30.510 7f515dc39140 1 Processor -- start >> 2018-12-12 22:35:30.510 7f515dc39140 0 mon.rio@-1(probing) e5 my rank is >> now 0 (was -1) >> 2018-12-12 22:35:30.510 7f515dc39140 1 -- 192.168.1.58:6789/0 >> shutdown_connections >> 2018-12-12 22:35:30.510 7f515dc39140 1 -- 192.168.1.58:6789/0 >> _send_message--> mon.1 192.168.1.59:6789/0 -- mon_probe(probe >> 376540c8-a362-41cc-9a58-9c8ceca0e4ee name rio) v6 -- ?+0 0x55d928525680 >> 2018-12-12 22:35:30.510 7f515dc39140 10 -- 192.168.1.58:6789/0 >> create_connect 192.168.1.59:6789/0, creating connection and registering >> 2018-12-12 22:35:30.510 7f515dc39140 10 -- 192.168.1.58:6789/0 >> >> 192.168.1.59:6789/0 conn(0x55d9281fc400 :-1 s=STATE_NONE pgs=0 cs=0 >> l=0)._connect csq=0 >> 2018-12-12 22:35:30.510 7f515dc39140 1 -- 192.168.1.58:6789/0 --> >> 192.168.1.59:6789/0 -- mon_probe(probe 376540c8-a362-41cc-9a58-9c8ceca0e4ee >> name rio) v6 -- 0x55d928525680 con 0 >> 2018-12-12 22:35:30.510 7f515dc39140 1 -- 192.168.1.58:6789/0 >> _send_message--> mon.2 192.168.1.65:6789/0 -- mon_probe(probe >> 376540c8-a362-41cc-9a58-9c8ceca0e4ee name rio) v6 -- ?+0 0x55d928525900 >> 2018-12-12 22:35:30.510 7f515dc39140 10 -- 192.168.1.58:6789/0 >> create_connect 192.168.1.65:6789/0, creating connection and registering >> 2018-12-12 22:35:30.510 7f515dc39140 10 -- 192.168.1.58:6789/0 >> >> 192.168.1.65:6789/0 conn(0x55d9281fca00 :-1 s=STATE_NONE pgs=0 cs=0 >> l=0)._connect csq=0 >> 2018-12-12 22:35:30.510 7f515dc39140 1 -- 192.168.1.58:6789/0 --> >> 192.168.1.65:6789/0 -- mon_probe(probe 376540c8-a362-41cc-9a58-9c8ceca0e4ee >> name rio) v6 -- 0x55d928525900 con 0 >> 2018-12-12 22:35:30.513 7f5143aec700 10 NetHandler generic_connect connect: >> (111) Connection refused >> 2018-12-12 22:35:30.513 7f51442ed700 10 NetHandler generic_connect connect: >> (111) Connection refused >> 2018-12-12 22:35:30.513 7f5144aee700 10 Infiniband send_msg sending: 0, >> 12894, 0, 0, fe800000000000007efe90fffe1e2524 >> 2018-12-12 22:35:30.513 7f5143aec700 1 RDMAStack connect try connecting >> failed. >> 2018-12-12 22:35:30.513 7f51442ed700 1 RDMAStack connect try connecting >> failed. >> 2018-12-12 22:35:30.513 7f5144aee700 10 -- - >> 192.168.1.88:6800/1638 >> conn(0x55d9281fbe00 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 >> l=0)._process_connection nonblock connect inprogress >> 2018-12-12 22:35:30.513 7f5143aec700 10 -- 192.168.1.58:6789/0 >> >> 192.168.1.59:6789/0 conn(0x55d9281fc400 :-1 s=STATE_CONNECTING pgs=0 cs=0 >> l=0).fault waiting 0.200000 >> 2018-12-12 22:35:30.513 7f51442ed700 10 -- 192.168.1.58:6789/0 >> >> 192.168.1.65:6789/0 conn(0x55d9281fca00 :-1 s=STATE_CONNECTING pgs=0 cs=0 >> l=0).fault waiting 0.200000 >> 2018-12-12 22:35:30.513 7f5144aee700 10 -- - >> 192.168.1.88:6800/1638 >> conn(0x55d9281fbe00 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0).handle_write >> 2018-12-12 22:35:30.513 7f5143aec700 10 -- 192.168.1.58:6789/0 >> >> 192.168.1.59:6789/0 conn(0x55d9281fc400 :-1 s=STATE_CONNECTING pgs=0 cs=0 >> l=0).handle_write >> 2018-12-12 22:35:30.513 7f51442ed700 10 -- 192.168.1.58:6789/0 >> >> 192.168.1.65:6789/0 conn(0x55d9281fca00 :-1 s=STATE_CONNECTING pgs=0 cs=0 >> l=0).handle_write >> 2018-12-12 22:35:30.513 7f5144aee700 5 Infiniband recv_msg recevd: 105, 0, >> 262144, 0, >> 2018-12-12 22:35:30.513 7f5144aee700 -1 RDMAConnectedSocketImpl activate >> failed to transition to RTR state: (113) No route to host >> 2018-12-12 22:35:30.515 7f5144aee700 -1 >> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.2/rpm/el7/BUILD/ceph-13.2.2/src/msg/async/rdma/RDMAConnectedSocketImpl.cc >> <http://rdmaconnectedsocketimpl.cc/>: In function 'void >> RDMAConnectedSocketImpl::handle_connection()' thread 7f5144aee700 time >> 2018-12-12 22:35:30.514762 >> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.2/rpm/el7/BUILD/ceph-13.2.2/src/msg/async/rdma/RDMAConnectedSocketImpl.cc >> <http://rdmaconnectedsocketimpl.cc/>: 224: FAILED assert(!r) >> >> ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) mimic >> (stable) >> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char >> const*)+0xff) [0x7f515506e6bf] >> 2: (()+0x285887) [0x7f515506e887] >> 3: (RDMAConnectedSocketImpl::handle_connection()+0x6f5) [0x7f51551b9655] >> 4: (EventCenter::process_events(unsigned int, >> std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0x695) >> [0x7f51551a76f5] >> 5: (()+0x3c15cc) [0x7f51551aa5cc] >> 6: (()+0x6afaef) [0x7f5155498aef] >> 7: (()+0x7e25) [0x7f5154396e25] >> 8: (clone()+0x6d) [0x7f5150cb7bad] >> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to >> interpret this. >> ------ceph-mon.rio.log--end------ >> >> -----ceph.conf---begin----- >> [client] >> rbd_cache = False >> rbd_cache_writethrough_until_flush = False >> #admin socket = /var/run/ceph/$cluster-$type.$id.$pid.$cctid.asok >> #log file = /var/log/ceph/ >> >> [global] >> ms_type = async+posix >> # set RDMA messaging just for the public or cluster network >> ms_public_type = async+rdma >> ms_cluster = async+rdma >> # >> # set a device name according to IB or ROCE device used, e.g. >> ms_async_rdma_device_name = mlx5_0 >> # >> # for better performance if using LUMINOUS 12.2.x release >> ms_async_rdma_polling_us = 0 >> ms_async_rdma_port_num=1 >> >> rgw_override_bucket_index_max_shards=1 >> auth client required = none >> auth cluster required = none >> auth service required = none >> auth supported = none >> cephx require signatures = False >> cephx sign messages = False >> cluster network = 192.168.1.0/24 >> debug asok = 0/0 >> debug auth = 0/0 >> debug bluefs = 0/0 >> debug bluestore = 0/0 >> debug buffer = 0/0 >> debug client = 0/0 >> debug context = 0/0 >> debug crush = 0/0 >> debug filer = 0/0 >> debug filestore = 0/0 >> debug finisher = 0/0 >> #debug hadoop = 0/0 -- doesn't work in E8 setup >> debug heartbeatmap = 0/0 >> debug journal = 0/0 >> debug journaler = 0/0 >> debug lockdep = 0/0 >> debug log = 0 >> debug mds = 0/0 >> debug mds_balancer = 0/0 >> debug mds_locker = 0/0 >> debug mds_log = 0/0 >> debug mds_log_expire = 0/0 >> debug mds_migrator = 0/0 >> debug mon = 0/0 >> debug monc = 0/0 >> debug ms = 10/10 >> #debug ms = 0/0 >> debug objclass = 0/0 >> debug objectcacher = 0/0 >> debug objecter = 0/0 >> debug optracker = 0/0 >> debug osd = 0/0 >> debug paxos = 0/0 >> debug perfcounter = 0/0 >> debug rados = 0/0 >> debug rbd = 0/0 >> debug rgw = 0/0 >> debug rocksdb = 0/0 >> debug throttle = 0/0 >> debug timer = 0/0 >> debug tp = 0/0 >> #debug zs = 0/0 -- doesn't work in E8 setup >> fsid = 376540c8-a362-41cc-9a58-9c8ceca0e4ee >> mon_host = 192.168.1.58,192.168.1.65,192.168.1.59 >> mon pg warn max per osd = 800 >> mon_allow_pool_delete = True >> mon_max_pg_per_osd = 800 >> ms type = async >> ms_crc_data = False >> #ms_crc_header = False -- broken! >> osd objectstore = bluestore >> osd_pool_default_size = 2 >> perf = True >> public network = 192.168.1.0/24 >> rocksdb_perf = True >> # Parameter not present in Micron's config, but introduced by ceph-deploy >> mon_initial_members = rio >> # The following param claims to reduce CPU usage; found it at >> # http://docs.ceph.com/docs/mimic/rados/configuration/ms-ref/ >> <http://docs.ceph.com/docs/mimic/rados/configuration/ms-ref/> >> ms nocrc = true >> #rbd_op_threads=4 >> [mon] >> mon_max_pool_pg_num = 166496 >> mon_osd_max_split_count = 10000 >> [osd] >> osd_min_pg_log_entries = 10 >> osd_max_pg_log_entries = 10 >> osd_pg_log_dups_tracked = 10 >> osd_pg_log_trim_min = 10 >> bluestore_cache_kv_max = 96G >> bluestore_cache_kv_ratio = 0.2 >> bluestore_cache_meta_ratio = 0.8 >> bluestore_cache_size_ssd = 7G >> bluestore_csum_type = none >> bluestore_extent_map_shard_max_size = 200 >> bluestore_extent_map_shard_min_size = 50 >> bluestore_extent_map_shard_target_size = 100 >> >> bluestore_rocksdb_options = >> compression=kNoCompression,max_write_buffer_number=64,min_write_buffer_number_to_merge=32,recycle_log_file_num=64,compaction_style=kCompactionStyleLevel,write_buffer_size=4MB,target_file_size_base=4MB,max_background_compactions=64,level0_file_num_compaction_trigger=64,level0_slowdown_writes_trigger=128,level0_stop_writes_trigger=256,max_bytes_for_level_base=6GB,compaction_threads=32,flusher_threads=8,compaction_readahead_size=2MB >> >> [client.rgw.sm26] >> rgw_frontends = "civetweb port=7480" >> >> #[osd.0] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae >> # >> #[osd.1] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae >> # >> #[osd.2] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae >> # >> #[osd.3] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae >> # >> #[osd.4] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae >> # >> #[osd.5] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae >> # >> #[osd.6] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae >> # >> #[osd.7] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae >> # >> #[osd.8] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae >> # >> #[osd.9] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae >> # >> #[osd.10] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae >> # >> #[osd.11] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae >> # >> #[osd.12] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae >> # >> #[osd.13] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae >> # >> #[osd.14] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae >> # >> #[osd.15] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae >> # >> ## BONJOVI1 >> # >> #[osd.16] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af >> # >> #[osd.17] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af >> # >> #[osd.18] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af >> # >> #[osd.19] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af >> # >> #[osd.20] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af >> # >> #[osd.21] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af >> # >> #[osd.22] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af >> # >> #[osd.23] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af >> # >> #[osd.24] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af >> # >> #[osd.25] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af >> # >> #[osd.26] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af >> # >> #[osd.27] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af >> # >> #[osd.28] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af >> # >> #[osd.29] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af >> # >> #[osd.30] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af >> # >> #[osd.31] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af >> # >> ## PRINCE >> # >> #[osd.32] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198 >> # >> #[osd.33] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198 >> #[osd.34] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198 >> # >> #[osd.35] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198 >> # >> #[osd.36] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198 >> # >> #[osd.37] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198 >> # >> #[osd.38] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198 >> # >> #[osd.39] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198 >> # >> #[osd.40] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198 >> # >> #[osd.41] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198 >> # >> #[osd.42] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198 >> # >> #[osd.43] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198 >> # >> #[osd.44] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198 >> # >> #[osd.45] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198 >> # >> #[osd.46] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198 >> # >> #[osd.47] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198 >> # >> # >> ## RINGO >> # >> #[osd.48] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e >> # >> #[osd.49] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e >> # >> #[osd.50] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e >> # >> #[osd.51] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e >> # >> #[osd.52] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e >> # >> #[osd.53] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e >> # >> #[osd.54] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e >> # >> #[osd.55] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e >> # >> #[osd.56] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e >> # >> #[osd.57] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e >> # >> #[osd.58] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e >> # >> #[osd.59] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e >> # >> #[osd.60] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e >> # >> #[osd.61] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e >> # >> #[osd.62] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e >> # >> #[osd.63] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e >> # >> #[mon.rio] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:013a >> # >> #[mon.salvador] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:013b >> # >> #[mon.medellin] >> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0141 >> >> -----ceph.conf---end----- >> >> >> -- > Michael Green > > > > > > > > > > > > -- > With best regards, > Vitaliy Filippov
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
