Hi Scott,
Would it be easy to patch bmi_mx to disable shared memory use until a
work-around is created?
Thanks,
Rob
Scott Atchley wrote:
On Jul 5, 2007, at 3:47 PM, Robert Latham wrote:
On Thu, Jul 05, 2007 at 02:37:02PM -0500, Glenn Johnson wrote:
The compute nodes will have the Myri-10G cards. The disks on the compute
nodes will make up a PVFS2 file system. The compute nodes will also be
PVFS2 clients. Data on this file system will consist of raw output from
calculations. The data files will not be immediately deleted but they
will not be backed up either. Users are expected to save post-processing
files in the home directory on the head node. The head node does not
have a Myri-10G card. I would like to be able to mount the PVFS2 file
system on the head node via Ethernet (Gb) if that is possible. I guess
what I am not sure about is if I can use the MX bmi for the I/O servers
and clients on the compute nodes and TCP for the client on the head
node.
Since the head node won't have MX, one approach might be to
"multi-home" the pvfs servers so they are listening on both the MX
interface (for the compute nodes) and on TCP (for the head node).
Pete does something like this for his Infiniband cluster.
Another approach, though one that is not likely to give great
performance, is to have the PVFS clients export the file system via
NFS to the head node.
==rob
Glenn,
For the nodes with Myri-10G cards that will be using bmi_mx, there is a
known issue if clients and servers are run on the same node _and_ the
client and server are run by different users. By default, MX tries to
use shared memory for traffic between processes on the same node. This
requires the processes to have the same owner. To avoid this issue, add
MX_DISABLE_SHMEM=1 to your environment until we have a longer-term
solution. Setting this variable will turn off MX shared memory and it
will cause MX to send local messages to the switch and back.
Scott
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
