This is actually an error propagating up from openIB, not pvfs. I've never
seen the error before, and I'm not sure if it is a fatal error or something
that we can handle inside pvfs, I'll have to look at the IB spec and see if we
can generate a patch for this.
[E 15:44:43.719270] Error: ib_check_cq: entry id 0x5c4e70 opcode RECV
error IBV_WC_WR_FLUSH_ERR.
Kyle
Tad Kollar wrote:
Pete Wyckoff wrote:
Have you been able to use, say, pvfs2-cp to put files into PVFS over
IB? That will help us know if it's a kernel problem or an IB
problem, perhaps.
After getting your reply I set up a test that used pvfs2-cp to copy a
2.5G file back and forth a total of 30 times. During that process,
pvfs2-cp generated these three errors, always during the read back from
the pvfs2 fs:
[E 15:44:43.719270] Error: ib_check_cq: entry id 0x5c4e70 opcode RECV
error IBV_WC_WR_FLUSH_ERR.
[E 15:44:43.924115] [bt] pvfs2-cp(error+0xca) [0x44a1ca]
[E 15:44:43.924161] [bt] pvfs2-cp [0x448dc3]
[E 15:44:43.924171] [bt] pvfs2-cp [0x4492c6]
[E 15:44:43.924179] [bt] pvfs2-cp(BMI_testcontext+0x151) [0x433371]
[E 15:44:43.924187] [bt] pvfs2-cp(PINT_thread_mgr_bmi_push+0x144)
[0x43c054]
[E 15:44:43.924195] [bt] pvfs2-cp(job_testcontext+0x15a) [0x43b87a]
[E 15:44:43.924204] [bt]
pvfs2-cp(PINT_client_state_machine_test+0x98) [0x40ff88]
[E 15:44:43.924211] [bt] pvfs2-cp(PVFS_sys_wait+0x63) [0x4103b3]
[E 15:44:43.924220] [bt] pvfs2-cp(PVFS_sys_io+0x6b) [0x41635b]
[E 15:44:43.924228] [bt] pvfs2-cp(main+0x372) [0x40d792]
[E 15:44:43.924236] [bt] /lib/libc.so.6(__libc_start_main+0xda)
[0x2aaaab0784ca]
[E 09:06:20.511281] Error: ib_check_cq: entry id 0x5e83f0 opcode RECV
error IBV_WC_WR_FLUSH_ERR.
[E 09:06:21.104063] [bt] pvfs2-cp(error+0xca) [0x44a1ca]
[E 09:06:21.104112] [bt] pvfs2-cp [0x448dc3]
[E 09:06:21.104120] [bt] pvfs2-cp [0x4492c6]
[E 09:06:21.104128] [bt] pvfs2-cp(BMI_testcontext+0x151) [0x433371]
[E 09:06:21.104136] [bt] pvfs2-cp(PINT_thread_mgr_bmi_push+0x144)
[0x43c054]
[E 09:06:21.104143] [bt] pvfs2-cp(job_testcontext+0x15a) [0x43b87a]
[E 09:06:21.104151] [bt]
pvfs2-cp(PINT_client_state_machine_test+0x98) [0x40ff88]
[E 09:06:21.104158] [bt] pvfs2-cp(PVFS_sys_wait+0x63) [0x4103b3]
[E 09:06:21.104165] [bt] pvfs2-cp(PVFS_sys_io+0x6b) [0x41635b]
[E 09:06:21.104173] [bt] pvfs2-cp(main+0x372) [0x40d792]
[E 09:06:21.104181] [bt] /lib/libc.so.6(__libc_start_main+0xda)
[0x2aaaab0784ca]
[E 09:09:46.596001] Error: ib_check_cq: entry id 0x5c4cc0 opcode RECV
error IBV_WC_WR_FLUSH_ERR.
[E 09:09:47.109736] [bt] pvfs2-cp(error+0xca) [0x44a1ca]
[E 09:09:47.109790] [bt] pvfs2-cp [0x448dc3]
[E 09:09:47.109799] [bt] pvfs2-cp [0x4492c6]
[E 09:09:47.109807] [bt] pvfs2-cp(BMI_testcontext+0x151) [0x433371]
[E 09:09:47.109816] [bt] pvfs2-cp(PINT_thread_mgr_bmi_push+0x144)
[0x43c054]
[E 09:09:47.109823] [bt] pvfs2-cp(job_testcontext+0x15a) [0x43b87a]
[E 09:09:47.109831] [bt]
pvfs2-cp(PINT_client_state_machine_test+0x98) [0x40ff88]
[E 09:09:47.109840] [bt] pvfs2-cp(PVFS_sys_wait+0x63) [0x4103b3]
[E 09:09:47.109847] [bt] pvfs2-cp(PVFS_sys_io+0x6b) [0x41635b]
[E 09:09:47.109856] [bt] pvfs2-cp(main+0x372) [0x40d792]
[E 09:09:47.109863] [bt] /lib/libc.so.6(__libc_start_main+0xda)
[0x2aaaab0784ca]
The other interesting thing to know is if you can recofigure PVFS to
use only TCP, then run your bonnie test and get the same error.
Except for IB testing, I've had TCP specified in the pvfs2tab and mount
options and haven't been able to disrupt it; is that sufficient or
should I remove all references to IB? I repeated the pvfs2-cp using TCP
and didn't receive any errors.
Tad
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
!DSPAM:45ffe85d58943491399889!
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
