Hi,
I'd appreciate the advice or insight anyone could provide on a pvfs2
configuration change I'm contemplating.
Specifically, I'd like to switch from using my current (congested)
Ethernet connections to using some Infiniband cards I have.
I have 'googled' the pvfs2 email list and one question about
multihoming didn't seem to have any result/thread. I didn't see
anything in the pvfs/doc/ folder, unless I missed it.
I'd appreciate any comments or suggestions about the options I should
set in the configuration set out below.
The setup:
--------------
I have three IO servers each with a 2-port Infiniband card. All ports
are connected to each other (no Infiniband switch). Using IPoIB I can
ping all IB ports from all machines, and running: "osmtest -f a",
indicates pass.
Each PVFS IO server server is also a PVFS client. One machine is the
metadata server.
The performance questions:
---------------------------------------
1) Is multi-homing in this setup likely to give benefits, or just headaches?
(a.k.a Is this a sane thing to contemplate?)
The configuration questions:
----------------------------------------
2) In pvfs2-conf.fs: should BMIModules refer to both tcp and ib to use IPoIB?
3) How should each IO server be "told" to listen on the two IP
addresses assigned to the 2 IB ports? Specifically:
a) Does HostID in the server config file need to be changed? If so,
b) Does the Alias in the pvfs2-conf.fs get changed?
c) Should TrustedPorts and TrustedNetwork values be set?
d) Should the port set on each IPoIB/IP address be different or the
same (I recall seeing an email about this but can't seem to find it
again...)
4) Given I will be using IPoIB I assume I still set tcp as the PVFS
protocol nominated in the HostID and Alias?
The configuration files are below.
Regards
Mark
[pvfs2-server.conf-frontend]
StorageSpace /state/partition1/pvfs2-storage-space
HostID "tcp://frontend:3334"
LogFile /var/log/pvfs2
[pvfs2-server.conf-compute-0-1]
StorageSpace /state/partition1/pvfs2-storage-space
HostID "tcp://pvfs2-compute-0-1:3334"
LogFile /var/log/pvfs2
[pvfs2-server.conf-compute-0-0]
StorageSpace /state/partition1/pvfs2-storage-space
HostID "tcp://pvfs2-compute-0-0:3334"
LogFile /var/log/pvfs2
[pvfs2-conf.fs]
<Defaults>
UnexpectedRequests 50
LogFile /var/log/pvfs2
EventLogging none
LogStamp datetime
BMIModules bmi_tcp
FlowModules flowproto_multiqueue
PerfUpdateInterval 1000
ServerJobBMITimeoutSecs 30
ServerJobFlowTimeoutSecs 30
ClientJobBMITimeoutSecs 300
ClientJobFlowTimeoutSecs 300
ClientRetryLimit 5
ClientRetryDelayMilliSecs 2000
</Defaults>
<Aliases>
Alias frontend tcp://frontend:3334
Alias pvfs2-compute-0-0 tcp://pvfs2-compute-0-0:3334
Alias pvfs2-compute-0-1 tcp://pvfs2-compute-0-1:3334
</Aliases>
<Filesystem>
Name pvfs2-fs
ID 1054160401
RootHandle 1048576
<MetaHandleRanges>
Range frontend 4-1431655767
</MetaHandleRanges>
<DataHandleRanges>
Range pvfs2-compute-0-0 1431655768-2863311531
Range pvfs2-compute-0-1 2863311532-4294967295
Range frontend 4294967295-6294967295
</DataHandleRanges>
<StorageHints>
TroveSyncMeta yes
TroveSyncData no
</StorageHints>
</Filesystem>
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
