This is what I did as suggested by Vu and it seems to be working. However, when I send 2MB IO, it gets broken into 512K+1MB+512K by SRP as seen on analyzer. I am just wondering what the logic is? On the other side, when we increase the srp_sg_tablesize beyond 256, we are seeing following message in /var/log/messages "Nov 29 21:17:50 p50 kernel: REJ reason 0x3" which indicates "IB_CM_REJ_NO_RESOURCES", so not sure how to get around to this problem to send larger IO than 1MB in one shot.
modprobe ib_srp srp_sg_tablesize=256 echo id_ext=200600A0B81138C9,max_sect=4096,ioc_guid=00a0b81112da0003,dgid=fe8 000000000000000a0b81112da0001,pkey=ffff,service_id=200600a0b81138c9> /sys/class/infiniband_srp/srp-mthca0-1/add_target -----Original Message----- From: chas williams - CONTRACTOR [mailto:[EMAIL PROTECTED] Sent: Friday, November 30, 2007 11:43 AM To: Kevin Harms Cc: Vu Pham; [EMAIL PROTECTED]; Batwara, Ashish Subject: Re: [ofa-general] IO Size more than 48K addtionally, you might need to echo 'blocks' > /sys/block/<device/queue/max_hw_segments to increase the size of the rdma segments. max_hw_segments doesnt exist on all kernels i think. In message <[EMAIL PROTECTED]>,Kevin Harms writ es: > > you may also have to go to /sys/block/sdX/queue and echo 1024 > >max_sectors_kb > if you use the srp_daemon you can also add: > a max_sect=2048 to /etc/srp_daemon.conf > >kevin > >On Nov 29, 2007, at 11:08 AM, Vu Pham wrote: > >> >>> Hi, >>> We are using OFED-1.2, and using xdd and some other tools, and >>> trying to >>> send 1/2MB IOs, but what we are seeing in analyzer traces, that >>> memory >>> descriptor in SRP command shows max. 48K which means 1MB I/Os has >>> broken >>> into smaller SRP request from initiator. >>> How can I have this I/O directly going to target? What parameter I >>> need >>> to change? >>> >>> >> >> module param srp_sg_tablesize (default is 12 ie. 12 x 4K = 48K) >> and/or >> max_sect=yyy in echo id_ext=xxx,...,max_sect=1024,service_id= > /sys/ >> class/infiniband_srp/... >> >> -vu >> >>> Thanks >>> Ashish >>> _______________________________________________ >>> general mailing list >>> [email protected] >>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >>> >>> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-gene >ral >>> >> >> _______________________________________________ >> general mailing list >> [email protected] >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >> >> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-gener >al >> > >_______________________________________________ >general mailing list >[email protected] >http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > >To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general >
_______________________________________________ general mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
