There are a number of LNET routing examples in the Lustre
documentation but the actual LNET topology requirements don't seem to
be spelled out. In the simplest terms, what are the actual
network topology requirements for the MDS, OSS's, and clients in
order for lustre to work correctly?
MDS NIDS
-------------------------
[EMAIL PROTECTED]
[EMAIL PROTECTED] (ip/ethernet) ----------------------------------------
|
OSS NIDS |
----------------------- |
[EMAIL PROTECTED] |
[EMAIL PROTECTED] (ipoib subnet) --------------------------- ipoib
gateway
|
IB Client NIDS |
----------------------- |
[EMAIL PROTECTED] |
|
Ethernet Client NIDS |
---------------------------- |
[EMAIL PROTECTED] ip/ethernet -------------------------------------------
There is no problem with the o2ib side of things. The IB network
is 10.13.24.0/21 so the clients, mds, and oss's are all on the same
"network".
The ethernet clients have no access to the 10.13.24.0 network.
They communicate with the MDS directly on the 10.13.16.0/22
network. However, the OSSs for the ethernet side all sit on their
own subnetted /30 network each of which is bridged by an IPoIB
gateway between the OSSs and the ethernet clients. So the MDS and
the clients talk to the OSSs through the IPoIB gateway and the OSSs
can talk to each other through the IPoIB Gateway (though on separate
subnets - there is a good reason for this, trust me :) ). So there
is complete connectivity in the "IP" sense.
However, after mounting the lustre file system on the ethernet
clients (which succeeds), the clients are always evicted immediately
following the obd_timeout period with a message such as...
Lustre: ufhpc-MDT0000: haven't heard from client
f6c6db8a-6fbc-6464-6261-af9ecfc0cb60 (at [EMAIL PROTECTED]) in 2462
seconds. I think it's dead, and I am evicting it.
Of course, once that happens, the client can no longer write to the
file system although it works fine for the "obd_timeout" period.
We don't completely understand this because all the players can talk
to each other via IP. Why would the MDT not be "hearing" from the
ethernet client? They are on the same IP network. It seems like
the problem is in the lnet topology? Do we really have to
introduce an lnet router even though we have complete IP connectivity
among the various components (mds, oss, clients)?
I'm hoping that someone more familiar with the LNET abstraction layer
can help us understand what the problem is.
Thanks,
charlie taylor
uf hpc center
_______________________________________________
Lustre-discuss mailing list
[email protected]
https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
