On Dec 25, 2006, at 9:59 AM, Pete Wyckoff wrote:
[EMAIL PROTECTED] wrote on Fri, 22 Dec 2006 15:45 -0600:
We stumbled across some hardware issues/limitations with our IB
hardware, and have 'devised' a way of working around these
limitations by increasing the sizes of memory region allocations
inside BMI, currently we're seeing chunks or blocks of 64KB being
allocated on the hardware level (this is what the firmware people
show us is allocated physically)
Is there a way we can tune these allocation calls to the IB drivers,
it should increase our ability to make large pinning requests if we
can use larger than 64KB, they suggest 2MB.
Are you aware of any specific memory region sizes that are allocated
in the BMI-IB layers?
Take a look at
http://www.beowulf-underground.org/pipermail/pvfs2-developers/
2006-October/002785.html
and the surrounding thread.
The patch that's discussed in there did get checked in, so recent
CVS HEAD will pin the entire buffer once if you try to send the
whole thing, rather than pinning multiple 64kB chunks of the buffer.
-- Pete
Excellent, thanks Pete! For some reason i thought that patch was
only for caching.
I built the latest pvfs-CVS, and am having problems with openib. I
originally thought it was a problem with having my client using CVS
and the server using the latest release, but rebuilt the server to be
on CVS head, and got this on the client:
p5l8:~# pvfs2-ls
[E 15:02:10.201683] Error: openib_new_connection: asked for 70 send
WRs on QP, got 0.
[E 15:02:10.201951] [bt] pvfs2-ls [0x100b659c]
[E 15:02:10.201982] [bt] pvfs2-ls [0x100b8698]
[E 15:02:10.202018] [bt] pvfs2-ls [0x100b5098]
[E 15:02:10.202054] [bt] pvfs2-ls [0x100b558c]
[E 15:02:10.202090] [bt] pvfs2-ls [0x100b22a8]
[E 15:02:10.202124] [bt] pvfs2-ls [0x100b2998]
[E 15:02:10.202161] [bt] pvfs2-ls [0x100b2ff0]
[E 15:02:10.202197] [bt] pvfs2-ls [0x10063cf8]
[E 15:02:10.202232] [bt] pvfs2-ls [0x10073d3c]
[E 15:02:10.202268] [bt] pvfs2-ls [0x1005d00c]
[E 15:02:10.202303] [bt] pvfs2-ls [0x10012ec0]
p5l8:~# pvfs2-ls --version
2.6.1pre1-2006-12-27-201055
Any thoughts on debugging this? I dont think I have problems with the
network unless something was sabotaged over the weekend. heh.
+=Kyle
_______________________________________________
Pvfs2-developers mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
