Hello Team, We are facing a lot of messages waiters related to waiting for conn rdmas < conn maxrdmas<https://www.mail-archive.com/search?l=gpfsug-discuss@spectrumscale.org&q=subject:%22Re%5C%3A+%5C%5Bgpfsug%5C-discuss%5C%5D+waiting+for+conn+rdmas+%3C+conn+maxrdmas%22&o=newest>
Is there some recommended settings to resolve this issue.? Our config for RDMA is as follows for 140 nodes(32 cores each) VERBS RDMA Configuration: Status : started Start time : Thu Stats reset time : Thu Dump time : Sun mmfs verbsRdma : enable mmfs verbsRdmaCm : disable mmfs verbsPorts : mlx4_0/1 mlx4_0/2 mmfs verbsRdmasPerNode : 3200 mmfs verbsRdmasPerNode (max) : 3200 mmfs verbsRdmasPerNodeOptimize : yes mmfs verbsRdmasPerConnection : 16 mmfs verbsRdmasPerConnection (max) : 16 mmfs verbsRdmaMinBytes : 16384 mmfs verbsRdmaRoCEToS : -1 mmfs verbsRdmaQpRtrMinRnrTimer : 18 mmfs verbsRdmaQpRtrPathMtu : 2048 mmfs verbsRdmaQpRtrSl : 0 mmfs verbsRdmaQpRtrSlDynamic : no mmfs verbsRdmaQpRtrSlDynamicTimeout : 10 mmfs verbsRdmaQpRtsRnrRetry : 6 mmfs verbsRdmaQpRtsRetryCnt : 6 mmfs verbsRdmaQpRtsTimeout : 18 mmfs verbsRdmaMaxSendBytes : 16777216 mmfs verbsRdmaMaxSendSge : 27 mmfs verbsRdmaSend : yes mmfs verbsRdmaSerializeRecv : no mmfs verbsRdmaSerializeSend : no mmfs verbsRdmaUseMultiCqThreads : yes mmfs verbsSendBufferMemoryMB : 1024 mmfs verbsLibName : libibverbs.so mmfs verbsRdmaCmLibName : librdmacm.so mmfs verbsRdmaMaxReconnectInterval : 60 mmfs verbsRdmaMaxReconnectRetries : -1 mmfs verbsRdmaReconnectAction : disable mmfs verbsRdmaReconnectThreads : 32 mmfs verbsHungRdmaTimeout : 90 ibv_fork_support : true Max connections : 196608 Max RDMA size : 16777216 Target number of vsend buffs : 16384 Initial vsend buffs per conn : 59 nQPs : 140 nCQs : 282 nCMIDs : 0 nDtoThreads : 2 nextIndex : 141 Number of Devices opened : 1 Device : mlx4_0 vendor_id : 713 Device vendor_part_id : 4099 Device mem register chunk : 8589934592 (0x200000000) Device max_sge : 32 Adjusted max_sge : 0 Adjusted max_sge vsend : 30 Device max_qp_wr : 16351 Device max_qp_rd_atom : 16 Open Connect Ports : 1 verbsConnectPorts[0] : mlx4_0/1/0 lid : 129 state : IBV_PORT_ACTIVE path_mtu : 2048 interface ID : 0xe41d2d030073b9d1 sendChannel.ib_channel : 0x7FA6CB816200 sendChannel.dtoThreadP : 0x7FA6CB821870 sendChannel.dtoThreadId : 12540 sendChannel.nFreeCq : 1 recvChannel.ib_channel : 0x7FA6CB81D590 recvChannel.dtoThreadP : 0x7FA6CB822BA0 recvChannel.dtoThreadId : 12541 recvChannel.nFreeCq : 1 ibv_cq : 0x7FA2724C81F8 ibv_cq.cqP : 0x0 ibv_cq.nEvents : 0 ibv_cq.contextP : 0x0 ibv_cq.ib_channel : 0x0 Thanks Tushar B Pathare MBA IT,BE IT Bigdata & GPFS Software Development & Databases Scientific Computing Bioinformatics Division Research "What ever the mind of man can conceive and believe, drill can query" Sidra Medical and Research Centre Sidra OPC Building Sidra Medical & Research Center PO Box 26999 Al Luqta Street Education City North Campus ​Qatar Foundation, Doha, Qatar Office 4003 3333 ext 37443 | M +974 74793547 tpath...@sidra.org<mailto:tpath...@sidra.org> | www.sidra.org<http://www.sidra.org/> Disclaimer: This email and its attachments may be confidential and are intended solely for the use of the individual to whom it is addressed. If you are not the intended recipient, any reading, printing, storage, disclosure, copying or any other action taken in respect of this e-mail is prohibited and may be unlawful. If you are not the intended recipient, please notify the sender immediately by using the reply function and then permanently delete what you have received. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Sidra Medical and Research Center.
_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss