Is RDMA officially supported? I'm asking because I recently tried to use DPDK
and it seems it's broken... i.e the code is there, but does not compile until I
fix cmake scripts, and after fixing the build OSDs just get segfaults and die
after processing something like 40-50 incoming packets.
Maybe RDMA is in the same state?
13 декабря 2018 г. 2:42:23 GMT+03:00, Michael Green <[email protected]> пишет:
>Sorry for bumping the thread. I refuse to believe there are no people
>on this list who have successfully enabled and run RDMA with Mimic. :)
>
>Mike
>
>> Hello collective wisdom,
>>
>> ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) mimic
>(stable) here.
>>
>> I have a working cluster here consisting of 3 monitor hosts, 64 OSD
>processes across 4 osd hosts, plus 2 MDSs, plus 2 MGRs. All of that is
>consumed by 10 client nodes.
>>
>> Every host in the cluster, including clients is
>> RHEL 7.5
>> Mellanox OFED 4.4-2.0.7.0
>> RoCE NICs are either MCX416A-CCAT or MCX414A-CCAT @ 50Gbit/sec
>> The NICs are all mlx5_0 port 1
>>
>> ring and ib_send_bw work fine both ways on any two nodes in the
>cluster.
>>
>> Full configuration of the cluster is pasted below, but RDMA related
>parameters are configured as following:
>>
>>
>> ms_public_type = async+rdma
>> ms_cluster = async+rdma
>> # Exclude clients for now
>> ms_type = async+posix
>>
>> ms_async_rdma_device_name = mlx5_0
>> ms_async_rdma_polling_us = 0
>> ms_async_rdma_port_num=1
>>
>> When I try to start MON, it immediately fails as below. Anybody has
>seen this or could give any pointers what to/where to look next?
>>
>>
>> ------ceph-mon.rio.log--begin------
>> 2018-12-12 22:35:30.011 7f515dc39140 0 set uid:gid to 167:167
>(ceph:ceph)
>> 2018-12-12 22:35:30.011 7f515dc39140 0 ceph version 13.2.2
>(02899bfda814146b021136e9d8e80eba494e1126) mimic (stable), process
>ceph-mon, pid 2129843
>> 2018-12-12 22:35:30.011 7f515dc39140 0 pidfile_write: ignore empty
>--pid-file
>> 2018-12-12 22:35:30.036 7f515dc39140 0 load: jerasure load: lrc
>load: isa
>> 2018-12-12 22:35:30.036 7f515dc39140 0 set rocksdb option
>compression = kNoCompression
>> 2018-12-12 22:35:30.036 7f515dc39140 0 set rocksdb option
>level_compaction_dynamic_level_bytes = true
>> 2018-12-12 22:35:30.036 7f515dc39140 0 set rocksdb option
>write_buffer_size = 33554432
>> 2018-12-12 22:35:30.036 7f515dc39140 0 set rocksdb option
>compression = kNoCompression
>> 2018-12-12 22:35:30.036 7f515dc39140 0 set rocksdb option
>level_compaction_dynamic_level_bytes = true
>> 2018-12-12 22:35:30.036 7f515dc39140 0 set rocksdb option
>write_buffer_size = 33554432
>> 2018-12-12 22:35:30.147 7f51442ed700 2 Event(0x55d927e95700
>nevent=5000 time_id=1).set_owner idx=1 owner=139987012998912
>> 2018-12-12 22:35:30.147 7f51442ed700 10 stack operator() starting
>> 2018-12-12 22:35:30.147 7f5143aec700 2 Event(0x55d927e95200
>nevent=5000 time_id=1).set_owner idx=0 owner=139987004606208
>> 2018-12-12 22:35:30.147 7f5144aee700 2 Event(0x55d927e95c00
>nevent=5000 time_id=1).set_owner idx=2 owner=139987021391616
>> 2018-12-12 22:35:30.147 7f5143aec700 10 stack operator() starting
>> 2018-12-12 22:35:30.147 7f5144aee700 10 stack operator() starting
>> 2018-12-12 22:35:30.147 7f515dc39140 0 starting mon.rio rank 0 at
>public addr 192.168.1.58:6789/0 at bind addr 192.168.1.58:6789/0
>mon_data /var/lib/ceph/mon/ceph-rio fsid
>376540c8-a362-41cc-9a58-9c8ceca0e4ee
>> 2018-12-12 22:35:30.147 7f515dc39140 10 -- - bind bind
>192.168.1.58:6789/0
>> 2018-12-12 22:35:30.147 7f515dc39140 10 -- - bind Network Stack is
>not ready for bind yet - postponed
>> 2018-12-12 22:35:30.147 7f515dc39140 0 starting mon.rio rank 0 at
>192.168.1.58:6789/0 mon_data /var/lib/ceph/mon/ceph-rio fsid
>376540c8-a362-41cc-9a58-9c8ceca0e4ee
>> 2018-12-12 22:35:30.148 7f515dc39140 0 mon.rio@-1(probing).mds e84
>new map
>> 2018-12-12 22:35:30.148 7f515dc39140 0 mon.rio@-1(probing).mds e84
>print_map
>> e84
>> enable_multiple, ever_enabled_multiple: 0,0
>> compat: compat={},rocompat={},incompat={1=base v0.20,2=client
>writeable ranges,3=default file layouts on dirs,4=dir inode in separate
>object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
>anchor table,9=file layout v2,10=snaprealm v2}
>> legacy client fscid: -1
>>
>> No filesystems configured
>> Standby daemons:
>>
>> 5906437: 192.168.1.152:6800/1077205146 'prince' mds.-1.0
>up:standby seq 2
>> 6284118: 192.168.1.59:6800/1266235911 'salvador' mds.-1.0
>up:standby seq 2
>>
>> 2018-12-12 22:35:30.148 7f515dc39140 0 mon.rio@-1(probing).osd
>e25894 crush map has features 288514051259236352, adjusting msgr
>requires
>> 2018-12-12 22:35:30.148 7f515dc39140 0 mon.rio@-1(probing).osd
>e25894 crush map has features 288514051259236352, adjusting msgr
>requires
>> 2018-12-12 22:35:30.148 7f515dc39140 0 mon.rio@-1(probing).osd
>e25894 crush map has features 1009089991638532096, adjusting msgr
>requires
>> 2018-12-12 22:35:30.148 7f515dc39140 0 mon.rio@-1(probing).osd
>e25894 crush map has features 288514051259236352, adjusting msgr
>requires
>> 2018-12-12 22:35:30.149 7f515dc39140 10 -- - create_connect
>192.168.1.88:6800/1638, creating connection and registering
>> 2018-12-12 22:35:30.149 7f515dc39140 10 -- - >>
>192.168.1.88:6800/1638 conn(0x55d9281fbe00 :-1 s=STATE_NONE pgs=0 cs=0
>l=0)._connect csq=0
>> 2018-12-12 22:35:30.149 7f515dc39140 10 -- - get_connection
>mgr.5894115 192.168.1.88:6800/1638 new 0x55d9281fbe00
>> 2018-12-12 22:35:30.150 7f515dc39140 1 -- - -->
>192.168.1.88:6800/1638 -- mgropen(unknown.rio) v3 -- 0x55d92844e000 con
>0
>> 2018-12-12 22:35:30.151 7f515dc39140 1 -- - start start
>> 2018-12-12 22:35:30.151 7f515dc39140 1 -- - start start
>> 2018-12-12 22:35:30.151 7f515dc39140 10 -- - ready -
>> 2018-12-12 22:35:30.151 7f515dc39140 10 -- - bind bind
>192.168.1.58:6789/0
>> 2018-12-12 22:35:30.151 7f515dc39140 10 Processor -- bind
>> 2018-12-12 22:35:30.154 7f5144aee700 1 Infiniband binding_port found
>active port 1
>> 2018-12-12 22:35:30.154 7f5144aee700 1 Infiniband init receive queue
>length is 4096 receive buffers
>> 2018-12-12 22:35:30.154 7f5144aee700 1 Infiniband init assigning:
>1024 send buffers
>> 2018-12-12 22:35:30.154 7f5144aee700 1 Infiniband init device allow
>4194303 completion entries
>> 2018-12-12 22:35:30.509 7f515dc39140 10 Processor -- bind bound to
>192.168.1.58:6789/0
>> 2018-12-12 22:35:30.509 7f515dc39140 1 -- 192.168.1.58:6789/0
>learned_addr learned my addr 192.168.1.58:6789/0
>> 2018-12-12 22:35:30.509 7f515dc39140 1 -- 192.168.1.58:6789/0
>_finish_bind bind my_inst.addr is 192.168.1.58:6789/0
>> 2018-12-12 22:35:30.510 7f515dc39140 10 -- - ready -
>> 2018-12-12 22:35:30.510 7f515dc39140 1 Processor -- start
>> 2018-12-12 22:35:30.510 7f515dc39140 0 mon.rio@-1(probing) e5 my
>rank is now 0 (was -1)
>> 2018-12-12 22:35:30.510 7f515dc39140 1 -- 192.168.1.58:6789/0
>shutdown_connections
>> 2018-12-12 22:35:30.510 7f515dc39140 1 -- 192.168.1.58:6789/0
>_send_message--> mon.1 192.168.1.59:6789/0 -- mon_probe(probe
>376540c8-a362-41cc-9a58-9c8ceca0e4ee name rio) v6 -- ?+0 0x55d928525680
>> 2018-12-12 22:35:30.510 7f515dc39140 10 -- 192.168.1.58:6789/0
>create_connect 192.168.1.59:6789/0, creating connection and registering
>> 2018-12-12 22:35:30.510 7f515dc39140 10 -- 192.168.1.58:6789/0 >>
>192.168.1.59:6789/0 conn(0x55d9281fc400 :-1 s=STATE_NONE pgs=0 cs=0
>l=0)._connect csq=0
>> 2018-12-12 22:35:30.510 7f515dc39140 1 -- 192.168.1.58:6789/0 -->
>192.168.1.59:6789/0 -- mon_probe(probe
>376540c8-a362-41cc-9a58-9c8ceca0e4ee name rio) v6 -- 0x55d928525680 con
>0
>> 2018-12-12 22:35:30.510 7f515dc39140 1 -- 192.168.1.58:6789/0
>_send_message--> mon.2 192.168.1.65:6789/0 -- mon_probe(probe
>376540c8-a362-41cc-9a58-9c8ceca0e4ee name rio) v6 -- ?+0 0x55d928525900
>> 2018-12-12 22:35:30.510 7f515dc39140 10 -- 192.168.1.58:6789/0
>create_connect 192.168.1.65:6789/0, creating connection and registering
>> 2018-12-12 22:35:30.510 7f515dc39140 10 -- 192.168.1.58:6789/0 >>
>192.168.1.65:6789/0 conn(0x55d9281fca00 :-1 s=STATE_NONE pgs=0 cs=0
>l=0)._connect csq=0
>> 2018-12-12 22:35:30.510 7f515dc39140 1 -- 192.168.1.58:6789/0 -->
>192.168.1.65:6789/0 -- mon_probe(probe
>376540c8-a362-41cc-9a58-9c8ceca0e4ee name rio) v6 -- 0x55d928525900 con
>0
>> 2018-12-12 22:35:30.513 7f5143aec700 10 NetHandler generic_connect
>connect: (111) Connection refused
>> 2018-12-12 22:35:30.513 7f51442ed700 10 NetHandler generic_connect
>connect: (111) Connection refused
>> 2018-12-12 22:35:30.513 7f5144aee700 10 Infiniband send_msg sending:
>0, 12894, 0, 0, fe800000000000007efe90fffe1e2524
>> 2018-12-12 22:35:30.513 7f5143aec700 1 RDMAStack connect try
>connecting failed.
>> 2018-12-12 22:35:30.513 7f51442ed700 1 RDMAStack connect try
>connecting failed.
>> 2018-12-12 22:35:30.513 7f5144aee700 10 -- - >>
>192.168.1.88:6800/1638 conn(0x55d9281fbe00 :-1 s=STATE_CONNECTING_RE
>pgs=0 cs=0 l=0)._process_connection nonblock connect inprogress
>> 2018-12-12 22:35:30.513 7f5143aec700 10 -- 192.168.1.58:6789/0 >>
>192.168.1.59:6789/0 conn(0x55d9281fc400 :-1 s=STATE_CONNECTING pgs=0
>cs=0 l=0).fault waiting 0.200000
>> 2018-12-12 22:35:30.513 7f51442ed700 10 -- 192.168.1.58:6789/0 >>
>192.168.1.65:6789/0 conn(0x55d9281fca00 :-1 s=STATE_CONNECTING pgs=0
>cs=0 l=0).fault waiting 0.200000
>> 2018-12-12 22:35:30.513 7f5144aee700 10 -- - >>
>192.168.1.88:6800/1638 conn(0x55d9281fbe00 :-1 s=STATE_CONNECTING_RE
>pgs=0 cs=0 l=0).handle_write
>> 2018-12-12 22:35:30.513 7f5143aec700 10 -- 192.168.1.58:6789/0 >>
>192.168.1.59:6789/0 conn(0x55d9281fc400 :-1 s=STATE_CONNECTING pgs=0
>cs=0 l=0).handle_write
>> 2018-12-12 22:35:30.513 7f51442ed700 10 -- 192.168.1.58:6789/0 >>
>192.168.1.65:6789/0 conn(0x55d9281fca00 :-1 s=STATE_CONNECTING pgs=0
>cs=0 l=0).handle_write
>> 2018-12-12 22:35:30.513 7f5144aee700 5 Infiniband recv_msg recevd:
>105, 0, 262144, 0,
>> 2018-12-12 22:35:30.513 7f5144aee700 -1 RDMAConnectedSocketImpl
>activate failed to transition to RTR state: (113) No route to host
>> 2018-12-12 22:35:30.515 7f5144aee700 -1
>/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.2/rpm/el7/BUILD/ceph-13.2.2/src/msg/async/rdma/RDMAConnectedSocketImpl.cc
><http://rdmaconnectedsocketimpl.cc/>: In function 'void
>RDMAConnectedSocketImpl::handle_connection()' thread 7f5144aee700 time
>2018-12-12 22:35:30.514762
>>
>/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.2/rpm/el7/BUILD/ceph-13.2.2/src/msg/async/rdma/RDMAConnectedSocketImpl.cc
><http://rdmaconnectedsocketimpl.cc/>: 224: FAILED assert(!r)
>>
>> ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) mimic
>(stable)
>> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>const*)+0xff) [0x7f515506e6bf]
>> 2: (()+0x285887) [0x7f515506e887]
>> 3: (RDMAConnectedSocketImpl::handle_connection()+0x6f5)
>[0x7f51551b9655]
>> 4: (EventCenter::process_events(unsigned int,
>std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l>
>>*)+0x695) [0x7f51551a76f5]
>> 5: (()+0x3c15cc) [0x7f51551aa5cc]
>> 6: (()+0x6afaef) [0x7f5155498aef]
>> 7: (()+0x7e25) [0x7f5154396e25]
>> 8: (clone()+0x6d) [0x7f5150cb7bad]
>> NOTE: a copy of the executable, or `objdump -rdS <executable>` is
>needed to interpret this.
>> ------ceph-mon.rio.log--end------
>>
>> -----ceph.conf---begin-----
>> [client]
>> rbd_cache = False
>> rbd_cache_writethrough_until_flush = False
>> #admin socket = /var/run/ceph/$cluster-$type.$id.$pid.$cctid.asok
>> #log file = /var/log/ceph/
>>
>> [global]
>> ms_type = async+posix
>> # set RDMA messaging just for the public or cluster network
>> ms_public_type = async+rdma
>> ms_cluster = async+rdma
>> #
>> # set a device name according to IB or ROCE device used, e.g.
>> ms_async_rdma_device_name = mlx5_0
>> #
>> # for better performance if using LUMINOUS 12.2.x release
>> ms_async_rdma_polling_us = 0
>> ms_async_rdma_port_num=1
>>
>> rgw_override_bucket_index_max_shards=1
>> auth client required = none
>> auth cluster required = none
>> auth service required = none
>> auth supported = none
>> cephx require signatures = False
>> cephx sign messages = False
>> cluster network = 192.168.1.0/24
>> debug asok = 0/0
>> debug auth = 0/0
>> debug bluefs = 0/0
>> debug bluestore = 0/0
>> debug buffer = 0/0
>> debug client = 0/0
>> debug context = 0/0
>> debug crush = 0/0
>> debug filer = 0/0
>> debug filestore = 0/0
>> debug finisher = 0/0
>> #debug hadoop = 0/0 -- doesn't work in E8 setup
>> debug heartbeatmap = 0/0
>> debug journal = 0/0
>> debug journaler = 0/0
>> debug lockdep = 0/0
>> debug log = 0
>> debug mds = 0/0
>> debug mds_balancer = 0/0
>> debug mds_locker = 0/0
>> debug mds_log = 0/0
>> debug mds_log_expire = 0/0
>> debug mds_migrator = 0/0
>> debug mon = 0/0
>> debug monc = 0/0
>> debug ms = 10/10
>> #debug ms = 0/0
>> debug objclass = 0/0
>> debug objectcacher = 0/0
>> debug objecter = 0/0
>> debug optracker = 0/0
>> debug osd = 0/0
>> debug paxos = 0/0
>> debug perfcounter = 0/0
>> debug rados = 0/0
>> debug rbd = 0/0
>> debug rgw = 0/0
>> debug rocksdb = 0/0
>> debug throttle = 0/0
>> debug timer = 0/0
>> debug tp = 0/0
>> #debug zs = 0/0 -- doesn't work in E8 setup
>> fsid = 376540c8-a362-41cc-9a58-9c8ceca0e4ee
>> mon_host = 192.168.1.58,192.168.1.65,192.168.1.59
>> mon pg warn max per osd = 800
>> mon_allow_pool_delete = True
>> mon_max_pg_per_osd = 800
>> ms type = async
>> ms_crc_data = False
>> #ms_crc_header = False -- broken!
>> osd objectstore = bluestore
>> osd_pool_default_size = 2
>> perf = True
>> public network = 192.168.1.0/24
>> rocksdb_perf = True
>> # Parameter not present in Micron's config, but introduced by
>ceph-deploy
>> mon_initial_members = rio
>> # The following param claims to reduce CPU usage; found it at
>> # http://docs.ceph.com/docs/mimic/rados/configuration/ms-ref/
><http://docs.ceph.com/docs/mimic/rados/configuration/ms-ref/>
>> ms nocrc = true
>> #rbd_op_threads=4
>> [mon]
>> mon_max_pool_pg_num = 166496
>> mon_osd_max_split_count = 10000
>> [osd]
>> osd_min_pg_log_entries = 10
>> osd_max_pg_log_entries = 10
>> osd_pg_log_dups_tracked = 10
>> osd_pg_log_trim_min = 10
>> bluestore_cache_kv_max = 96G
>> bluestore_cache_kv_ratio = 0.2
>> bluestore_cache_meta_ratio = 0.8
>> bluestore_cache_size_ssd = 7G
>> bluestore_csum_type = none
>> bluestore_extent_map_shard_max_size = 200
>> bluestore_extent_map_shard_min_size = 50
>> bluestore_extent_map_shard_target_size = 100
>>
>> bluestore_rocksdb_options =
>compression=kNoCompression,max_write_buffer_number=64,min_write_buffer_number_to_merge=32,recycle_log_file_num=64,compaction_style=kCompactionStyleLevel,write_buffer_size=4MB,target_file_size_base=4MB,max_background_compactions=64,level0_file_num_compaction_trigger=64,level0_slowdown_writes_trigger=128,level0_stop_writes_trigger=256,max_bytes_for_level_base=6GB,compaction_threads=32,flusher_threads=8,compaction_readahead_size=2MB
>>
>> [client.rgw.sm26]
>> rgw_frontends = "civetweb port=7480"
>>
>> #[osd.0]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
>> #
>> #[osd.1]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
>> #
>> #[osd.2]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
>> #
>> #[osd.3]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
>> #
>> #[osd.4]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
>> #
>> #[osd.5]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
>> #
>> #[osd.6]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
>> #
>> #[osd.7]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
>> #
>> #[osd.8]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
>> #
>> #[osd.9]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
>> #
>> #[osd.10]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
>> #
>> #[osd.11]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
>> #
>> #[osd.12]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
>> #
>> #[osd.13]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
>> #
>> #[osd.14]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
>> #
>> #[osd.15]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
>> #
>> ## BONJOVI1
>> #
>> #[osd.16]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
>> #
>> #[osd.17]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
>> #
>> #[osd.18]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
>> #
>> #[osd.19]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
>> #
>> #[osd.20]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
>> #
>> #[osd.21]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
>> #
>> #[osd.22]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
>> #
>> #[osd.23]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
>> #
>> #[osd.24]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
>> #
>> #[osd.25]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
>> #
>> #[osd.26]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
>> #
>> #[osd.27]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
>> #
>> #[osd.28]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
>> #
>> #[osd.29]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
>> #
>> #[osd.30]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
>> #
>> #[osd.31]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
>> #
>> ## PRINCE
>> #
>> #[osd.32]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
>> #
>> #[osd.33]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
>> #[osd.34]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
>> #
>> #[osd.35]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
>> #
>> #[osd.36]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
>> #
>> #[osd.37]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
>> #
>> #[osd.38]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
>> #
>> #[osd.39]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
>> #
>> #[osd.40]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
>> #
>> #[osd.41]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
>> #
>> #[osd.42]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
>> #
>> #[osd.43]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
>> #
>> #[osd.44]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
>> #
>> #[osd.45]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
>> #
>> #[osd.46]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
>> #
>> #[osd.47]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
>> #
>> #
>> ## RINGO
>> #
>> #[osd.48]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
>> #
>> #[osd.49]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
>> #
>> #[osd.50]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
>> #
>> #[osd.51]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
>> #
>> #[osd.52]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
>> #
>> #[osd.53]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
>> #
>> #[osd.54]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
>> #
>> #[osd.55]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
>> #
>> #[osd.56]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
>> #
>> #[osd.57]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
>> #
>> #[osd.58]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
>> #
>> #[osd.59]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
>> #
>> #[osd.60]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
>> #
>> #[osd.61]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
>> #
>> #[osd.62]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
>> #
>> #[osd.63]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
>> #
>> #[mon.rio]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:013a
>> #
>> #[mon.salvador]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:013b
>> #
>> #[mon.medellin]
>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0141
>>
>> -----ceph.conf---end-----
>>
>>
>> --
>Michael Green
--
With best regards,
Vitaliy Filippov_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
