I've been running across problems when trying to do rdma's of large
files over IB on pvfs2, where the mop_id somehow becomes invalid or
broken whenever attempting to write sizes over 512MB. (sometimes occurs
on 256MB, however, is not reproducable for sizes <256MB)
- error is in encourage_send-incoming_cts, supposedly while we:
/*
* Look through this CTS message to determine the owning sq. Works
* using the mop_id which was sent during the RTS, now returned to us.
*/
Setup is: 1Meta+Data Server, w/ filesystem mounted to local-disk.
Tests are using pvfs2-cp with test files of 512MB: (problems are
reproducable on two (all) clients that I've been testing with)
Output:
[EMAIL PROTECTED]:~/pvfs2$ inst1.4/bin/pvfs2-cp -t /tmp/bigfile3 /mnt/pvfs2/
[E 17:58:03.909494] Error: encourage_send_incoming_cts: mop_id 1962 in
CTS message not found.
[EMAIL PROTECTED]:~/pvfs2$ inst1.4/bin/pvfs2-cp -t /tmp/bigfile2 /mnt/pvfs2
[E 17:59:30.578328] Error: encourage_send_incoming_cts: mop_id 2009 in
CTS message not found.
Why would we be getting erroneous mop_id's in the cts??
On a more positive / successful run, with a smaller file (128MB) yields
much better results:
[EMAIL PROTECTED]:~/pvfs2$ inst1.4/bin/pvfs2-cp -t /tmp/bigfile /mnt/pvfs2
Wrote 134217728 bytes in 0.463656 seconds. 276.066770 MB/seconds
On another note:
Pete, Is there any particular reasoning for your selection of the
qp_init_attributes in ib_new_connection() [setup.c:119] ?
And would there be any way to figure out if we're using all of the
'resources' we've specified using those settings?
(I've had to change several of them to get them to work with my HCA's)
thanks,
~Kyle
--
Kyle Schochenmaier
[EMAIL PROTECTED]
Research Assistant, Dr. Brett Bode
AmesLab - US Dept.Energy
Scalable Computing Laboratory
_______________________________________________
Pvfs2-developers mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
