Sam and I looked into a problem we found with the noncontig-test
that I'm using as one of my benchmarks in my suite.
Test setup:
pvfs2-fs: MX on 4 data servers, 5th server is the client. (CVS Head)
If I run the test using MX, it will fail, but with TCP, the test
completes, we had originally thought that this was a problem in
the pint-request code (as the log will indicate) but I'm
wondering now why it would fail using a different transport.. To
clear up the obvious problems, I've run other benchmarks using
the same setup, before and after this error shows up and those
all run to completion just fine on both mx and tcp.
Any ideas where to start with this?
thanks,
Kyle
__Output__
TCP:
[EMAIL PROTECTED]:~/framework/noncontig-test/noncontig$ mpirun -np
1 ./noncontig -fname pvfs2://tmp/pvfs2/blah -fsize 1 -timing
========= Parameter space dump =========
filename: pvfs2://tmp/pvfs2/blah ionodes
file size (MB): 1 buffer size 0
vector length: 10 element count: 1 vector count: 0
striping factor: 0 striping size: -1 collective buffer size: 0
loops: 1 displacement 0
========= Dump done =========
#* no verification possible!
# testing noncontiguous in memory, noncontiguous in file using
independent I/O
# vector count = 26214 - access count = 26214
write bandwidth (min/max/acc [MB/s]) : 0.331 / 0.331 / 0.331
read bandwidth (min/max/acc [MB/s]) : 0.370 / 0.370 / 0.370
file size: 1024kB size per process: 1023kB
# testing noncontiguous in memory, contiguous in file using
independent I/O
# vector count = 26214 - access count = 26214
write bandwidth (min/max/acc [MB/s]) : 0.692 / 0.692 / 0.692
read bandwidth (min/max/acc [MB/s]) : 0.766 / 0.766 / 0.766
file size: 1023kB size per process: 1023kB
# testing contiguous in memory, noncontiguous in file using
independent I/O
# vector count = 26214 - access count = 26214
write bandwidth (min/max/acc [MB/s]) : 0.348 / 0.348 / 0.348
read bandwidth (min/max/acc [MB/s]) : 0.392 / 0.392 / 0.392
file size: 1024kB size per process: 1023kB
MX:
[EMAIL PROTECTED]:~/framework/noncontig-test/noncontig$ `mpirun -np
1 ./noncontig -fname pvfs2://tmp/pvfs2/blah -fsize 1 &> mx_output`
========= Parameter space dump =========
filename: pvfs2://tmp/pvfs2/blah ionodes
file size (MB): 1 buffer size 0
vector length: 10 element count: 1 vector count: 0
striping factor: 0 striping size: -1 collective buffer size: 0
loops: 1 displacement 0
========= Dump done =========
#* no verification possible!
# testing noncontiguous in memory, noncontiguous in file using
independent I/O
# vector count = 26214 - access count = 26214
[E 13:39:06.029976] src/io/description/pint-request.c line 95:
PINT_process_requ
est: no segments or bytes requested!
[E 13:39:06.030497] [bt] ./noncontig [0x4cd655]
[E 13:39:06.030555] [bt] ./noncontig [0x4b2e01]
[E 13:39:06.030608] [bt] ./noncontig [0x4ae8f1]
[E 13:39:06.030658] [bt] ./noncontig [0x507b62]
[E 13:39:06.030707] [bt] ./noncontig [0x5080dd]
[E 13:39:06.030756] [bt] ./noncontig [0x507e2f]
[E 13:39:06.030806] [bt] ./noncontig [0x4a5030]
[E 13:39:06.030854] [bt] ./noncontig [0x4ae202]
[E 13:39:06.030903] [bt] ./noncontig [0x4ae2d5]
[E 13:39:06.030952] [bt] ./noncontig [0x479ab0]
[E 13:39:06.031001] [bt] ./noncontig [0x41df43]
[E 13:39:06.031072] PVFS_isys_io call: Invalid argument
[0] Error -524286 in MPI_File_write
Undefined dynamic error code
[E 13:39:06.067249] Warning: non PVFS2 error code (22):
[E 13:39:06.067468] Send immediately failed: Invalid argument
[E 13:39:06.067525] Send error: cancelling recv.
[E 13:39:06.067599] Warning: non PVFS2 error code (22):
[E 13:39:06.067651] msgpair failed, will retry: Invalid argument
[E 13:39:06.067706] *** msgpairarray_completion_fn: msgpair to
server mx://bb15:
0:3 failed: Invalid argument
[E 13:39:06.067755] *** Non-BMI failure.
[E 13:39:06.074742] Warning: non PVFS2 error code (22):
[E 13:39:06.074795] Send immediately failed: Invalid argument
[E 13:39:06.074843] Send error: cancelling recv.
[E 13:39:06.074900] Warning: non PVFS2 error code (22):
[E 13:39:06.074948] msgpair failed, will retry: Invalid argument
[E 13:39:06.074998] *** msgpairarray_completion_fn: msgpair to
server mx://bb15:
0:3 failed: Invalid argument
[E 13:39:06.075046] *** Non-BMI failure.
[E 13:39:06.075396] Warning: non PVFS2 error code (22):
[E 13:39:06.075447] Send immediately failed: Invalid argument
[E 13:39:06.075493] Send error: cancelling recv.
[E 13:39:06.075551] Warning: non PVFS2 error code (22):
[E 13:39:06.075599] msgpair failed, will retry: Invalid argument
[E 13:39:06.075649] *** msgpairarray_completion_fn: msgpair to
server mx://bb15:
