Matthew, Sam-
FYI, using jumbo frames is not necessarily that simple. The IBM file
servers that came with our BG/L don't support jumbo frames on the
internal NICs. Ours are IBM x346 type 8840, and the built-in
Broadcom NICs couldn't handle jumbo frames. I imagine other xSeries
boxes with integrated Broadcom NICs may have similar issues. We
ended up buying PCI network cards in order to implement jumbo frames
in our environment. Also, you'll need to make sure your network
switch can handle jumbo frames (ours is a Force10, don't know the
exact model off the top of my head but it supports jumbo frames).
The other thing you need to be aware of is that switching to jumbo
frames is an all-or-nothing proposition; if you do it, you'll have to
do it for *all* of the hardware on the involved network segment. You
can't just change a couple of servers.
I'm Cc:ing a couple of folks at Argonne who worked on getting jumbo
frames working for our environment; they might be able to warn you of
any other gotchas. We're using 8000 byte frames, but if I were
starting from scratch I'd try something closer to 8300 so that an
entire 8192-byte NFS packet can fit in a single frame (avoiding
fragmentation if you're using an 8192 byte NFS rsize/wsize). Note,
8300 is just a ballparck guess that I haven't been able to confirm.
Be warned -- in our environment, we started to have problems when we
got close to 9000 byte frames, so don't go too high.
-Andrew Cherry
BG/L Support
Argonne National Laboratory
On Apr 20, 2007, at 5:02 PM, Sam Lang wrote:
Hi Matthew,
I think the version of PVFS in the Zepto release is pvfs2-1.5.1.
Besides some performance improvements in the latest release
(pvfs-2.6.3), there was a specific bugfix made in PVFS for largish
mpi-io jobs. If you could try the latest (at http://
www.pvfs.org/), it would help us to verify that you're not running
into the same problem.
Regarding config options for PVFS on BGL, make sure you have jumbo
frames enabled, i.e.
ifconfig eth0 mtu 8000 up
Also, you should probably set the tcp buffer sizes explicitly in
the pvfs config file, fs.conf:
<Defaults>
...
TCPBufferSend 524288
TCPBufferReceive 1048576
</Defaults>
You might also see better performance with an alternative trove
method for doing disk io:
<StorageHints>
...
TroveMethod alt-aio
</StorageHints>
Thanks,
-sam
On Apr 20, 2007, at 4:25 PM, Matthew Woitaszek wrote:
Good afternoon,
Michael Oberg and I are attempting to get PVFS2 working on NCAR's
1-rack
BlueGene/L system using ZeptoOS. We ran into a snag at over 8 BG/L
I/O nodes
(>256 compute nodes).
We've been using the mpi-io-test program shipped with PVFS2 to
test the
system. For cases up to and including 8 I/O nodes (256 coprocessor
or 512
virtual node mode tasks), everything works fine. Larger jobs fail
with file
not found error messages, such as:
MPI_File_open: File does not exist, error stack:
ADIOI_BGL_OPEN(54): File /pvfs2/mattheww/_file_0512_co does not
exist
The file is created on the PVFS2 filesystem and has a zero-byte
size. We've
run the tests with 512 tasks on 256 nodes, and it successfully
created a
8589934592-byte file. Going to 257 nodes fails.
Has anyone seen this behavior before? Are there any PVFS2 server
or client
configuration options that you would recommend for a BG/L
installation like
this?
Thanks for your time,
Matthew
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
