Pete Wyckoff wrote: > Have you been able to use, say, pvfs2-cp to put files into PVFS over > IB? That will help us know if it's a kernel problem or an IB > problem, perhaps. > After getting your reply I set up a test that used pvfs2-cp to copy a 2.5G file back and forth a total of 30 times. During that process, pvfs2-cp generated these three errors, always during the read back from the pvfs2 fs:
[E 15:44:43.719270] Error: ib_check_cq: entry id 0x5c4e70 opcode RECV error IBV_WC_WR_FLUSH_ERR. [E 15:44:43.924115] [bt] pvfs2-cp(error+0xca) [0x44a1ca] [E 15:44:43.924161] [bt] pvfs2-cp [0x448dc3] [E 15:44:43.924171] [bt] pvfs2-cp [0x4492c6] [E 15:44:43.924179] [bt] pvfs2-cp(BMI_testcontext+0x151) [0x433371] [E 15:44:43.924187] [bt] pvfs2-cp(PINT_thread_mgr_bmi_push+0x144) [0x43c054] [E 15:44:43.924195] [bt] pvfs2-cp(job_testcontext+0x15a) [0x43b87a] [E 15:44:43.924204] [bt] pvfs2-cp(PINT_client_state_machine_test+0x98) [0x40ff88] [E 15:44:43.924211] [bt] pvfs2-cp(PVFS_sys_wait+0x63) [0x4103b3] [E 15:44:43.924220] [bt] pvfs2-cp(PVFS_sys_io+0x6b) [0x41635b] [E 15:44:43.924228] [bt] pvfs2-cp(main+0x372) [0x40d792] [E 15:44:43.924236] [bt] /lib/libc.so.6(__libc_start_main+0xda) [0x2aaaab0784ca] [E 09:06:20.511281] Error: ib_check_cq: entry id 0x5e83f0 opcode RECV error IBV_WC_WR_FLUSH_ERR. [E 09:06:21.104063] [bt] pvfs2-cp(error+0xca) [0x44a1ca] [E 09:06:21.104112] [bt] pvfs2-cp [0x448dc3] [E 09:06:21.104120] [bt] pvfs2-cp [0x4492c6] [E 09:06:21.104128] [bt] pvfs2-cp(BMI_testcontext+0x151) [0x433371] [E 09:06:21.104136] [bt] pvfs2-cp(PINT_thread_mgr_bmi_push+0x144) [0x43c054] [E 09:06:21.104143] [bt] pvfs2-cp(job_testcontext+0x15a) [0x43b87a] [E 09:06:21.104151] [bt] pvfs2-cp(PINT_client_state_machine_test+0x98) [0x40ff88] [E 09:06:21.104158] [bt] pvfs2-cp(PVFS_sys_wait+0x63) [0x4103b3] [E 09:06:21.104165] [bt] pvfs2-cp(PVFS_sys_io+0x6b) [0x41635b] [E 09:06:21.104173] [bt] pvfs2-cp(main+0x372) [0x40d792] [E 09:06:21.104181] [bt] /lib/libc.so.6(__libc_start_main+0xda) [0x2aaaab0784ca] [E 09:09:46.596001] Error: ib_check_cq: entry id 0x5c4cc0 opcode RECV error IBV_WC_WR_FLUSH_ERR. [E 09:09:47.109736] [bt] pvfs2-cp(error+0xca) [0x44a1ca] [E 09:09:47.109790] [bt] pvfs2-cp [0x448dc3] [E 09:09:47.109799] [bt] pvfs2-cp [0x4492c6] [E 09:09:47.109807] [bt] pvfs2-cp(BMI_testcontext+0x151) [0x433371] [E 09:09:47.109816] [bt] pvfs2-cp(PINT_thread_mgr_bmi_push+0x144) [0x43c054] [E 09:09:47.109823] [bt] pvfs2-cp(job_testcontext+0x15a) [0x43b87a] [E 09:09:47.109831] [bt] pvfs2-cp(PINT_client_state_machine_test+0x98) [0x40ff88] [E 09:09:47.109840] [bt] pvfs2-cp(PVFS_sys_wait+0x63) [0x4103b3] [E 09:09:47.109847] [bt] pvfs2-cp(PVFS_sys_io+0x6b) [0x41635b] [E 09:09:47.109856] [bt] pvfs2-cp(main+0x372) [0x40d792] [E 09:09:47.109863] [bt] /lib/libc.so.6(__libc_start_main+0xda) [0x2aaaab0784ca] > The other interesting thing to know is if you can recofigure PVFS to > use only TCP, then run your bonnie test and get the same error. > Except for IB testing, I've had TCP specified in the pvfs2tab and mount options and haven't been able to disrupt it; is that sufficient or should I remove all references to IB? I repeated the pvfs2-cp using TCP and didn't receive any errors. Tad _______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
