Re: [Pvfs2-developers] [Fwd: Molpro details]

Murali Vilayannur Sat, 11 Nov 2006 02:45:26 -0800

Hi Brian,
Not sure if anyone got to this email.
Sorry if it got lost ...
On how many client-nodes was Molpro running on? How many servers?

It looks like MolPro is not an open-source app for us to reproduce theerror though... :(

Is this with the client-side vfs patches that we had mailed to youseparately along with the upped timeout values for flow? Or was it plain1.5.1?

Have you had any other problems subsequently with this workload?

Hopefully, the new release will be able to handle I/O timeout issues alittle bit better than 1.5.1

although I suspect this might be a different unrelated problem/bug.
thanks,
Murali

We are seeing another application having issues accessing pvfs2 spacevia the client side mount via the kernel module. I'm forwarding adescription of things and looking for help on what info to gather toadd to this report. What I have so far is the client and server logoutput from during her runs of these test cases.
Client syslog messages:
Oct 10 12:40:58 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017704, -- returning -2Oct 10 12:40:59 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017704, -- returning -2Oct 10 12:40:59 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017703, -- returning -2Oct 10 12:40:59 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017703, -- returning -2Oct 10 12:40:59 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017702, -- returning -2Oct 10 12:41:00 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017702, -- returning -2Oct 10 12:41:00 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017700, -- returning -2Oct 10 12:41:02 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017698, -- returning -2Oct 10 12:41:04 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017697, -- returning -2Oct 10 12:41:06 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017697, -- returning -2Oct 10 12:41:06 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017696, -- returning -2Oct 10 12:41:07 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017695, -- returning -2Oct 10 12:41:07 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017695, -- returning -2Oct 10 12:41:07 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017694, -- returning -2Oct 10 12:41:08 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017694, -- returning -2Oct 10 12:41:08 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017693, -- returning -2Oct 10 12:41:09 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017693, -- returning -2Oct 10 12:41:09 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017692, -- returning -2Oct 10 12:41:09 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017692, -- returning -2Oct 10 12:41:09 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017691, -- returning -2Oct 10 12:41:09 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017691, -- returning -2Oct 10 12:41:09 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017690, -- returning -2Oct 10 12:41:10 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017689, -- returning -2Oct 10 12:41:11 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017689, -- returning -2Oct 10 12:41:11 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017688, -- returning -2Oct 10 12:41:11 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017687, -- returning -2Oct 10 12:41:12 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017686, -- returning -2Oct 10 12:41:12 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017685, -- returning -2Oct 10 12:41:13 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017684, -- returning -2Oct 10 12:41:13 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017684, -- returning -2Oct 10 12:41:13 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017683, -- returning -2Oct 10 12:41:14 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017683, -- returning -2Oct 10 12:41:14 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017682, -- returning -2Oct 10 12:41:14 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017681, -- returning -2Oct 10 12:41:15 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017680, -- returning -2Oct 10 12:41:16 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017680, -- returning -2Oct 10 12:41:16 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017679, -- returning -2Oct 10 12:41:17 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017679, -- returning -2Oct 10 12:41:17 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017678, -- returning -2Oct 10 12:41:18 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017678, -- returning -2Oct 10 12:41:18 la099.landscapearch.arches kernel: pvfs2_file_write:error writing to handle 1017677, -- returning -2
Server pvfs2 logs showing last 40 lines from each of the 12 servers:
[EMAIL PROTECTED] snapshot]# tail -40 ff*/var/log/pvfs*
==> ff001/var/log/pvfs2-server.log <==
[E 01:28:36.025839] TROVE:DBPF:Berkeley DB: keyval_db->put:DB_KEYEXIST: Key/data pair already exists[E 01:28:36.039854] TROVE:DBPF:Berkeley DB: keyval_db->put:DB_KEYEXIST: Key/data pair already exists[E 01:28:36.042510] TROVE:DBPF:Berkeley DB: keyval_db->put:DB_KEYEXIST: Key/data pair already exists[E 01:28:36.269407] TROVE:DBPF:Berkeley DB: keyval_db->put:DB_KEYEXIST: Key/data pair already exists[E 01:28:36.269550] TROVE:DBPF:Berkeley DB: keyval_db->put:DB_KEYEXIST: Key/data pair already exists[E 01:28:36.269672] TROVE:DBPF:Berkeley DB: keyval_db->put:DB_KEYEXIST: Key/data pair already exists[E 01:28:36.565606] TROVE:DBPF:Berkeley DB: keyval_db->put:DB_KEYEXIST: Key/data pair already exists[E 01:28:36.565808] TROVE:DBPF:Berkeley DB: keyval_db->put:DB_KEYEXIST: Key/data pair already exists[E 01:28:36.566065] TROVE:DBPF:Berkeley DB: keyval_db->put:DB_KEYEXIST: Key/data pair already exists[E 01:28:37.320512] TROVE:DBPF:Berkeley DB: keyval_db->put:DB_KEYEXIST: Key/data pair already exists[E 01:28:37.391505] TROVE:DBPF:Berkeley DB: keyval_db->put:DB_KEYEXIST: Key/data pair already exists[E 01:28:37.391707] TROVE:DBPF:Berkeley DB: keyval_db->put:DB_KEYEXIST: Key/data pair already exists[E 01:28:37.605731] TROVE:DBPF:Berkeley DB: keyval_db->put:DB_KEYEXIST: Key/data pair already exists[E 01:28:37.616642] TROVE:DBPF:Berkeley DB: keyval_db->put:DB_KEYEXIST: Key/data pair already exists[E 01:28:37.643355] TROVE:DBPF:Berkeley DB: keyval_db->put:DB_KEYEXIST: Key/data pair already exists[E 01:28:37.830465] TROVE:DBPF:Berkeley DB: keyval_db->put:DB_KEYEXIST: Key/data pair already exists[E 01:28:37.831224] TROVE:DBPF:Berkeley DB: keyval_db->put:DB_KEYEXIST: Key/data pair already exists[E 01:28:37.831336] TROVE:DBPF:Berkeley DB: keyval_db->put:DB_KEYEXIST: Key/data pair already exists[E 01:28:37.925518] TROVE:DBPF:Berkeley DB: keyval_db->put:DB_KEYEXIST: Key/data pair already exists[E 01:28:37.925677] TROVE:DBPF:Berkeley DB: keyval_db->put:DB_KEYEXIST: Key/data pair already exists[E 01:28:37.925791] TROVE:DBPF:Berkeley DB: keyval_db->put:DB_KEYEXIST: Key/data pair already exists[E 01:28:38.268132] TROVE:DBPF:Berkeley DB: keyval_db->put:DB_KEYEXIST: Key/data pair already exists[E 01:28:38.268692] TROVE:DBPF:Berkeley DB: keyval_db->put:DB_KEYEXIST: Key/data pair already exists[E 01:28:38.291768] TROVE:DBPF:Berkeley DB: keyval_db->put:DB_KEYEXIST: Key/data pair already exists[E 01:28:38.511212] TROVE:DBPF:Berkeley DB: keyval_db->put:DB_KEYEXIST: Key/data pair already exists[E 01:28:38.529310] TROVE:DBPF:Berkeley DB: keyval_db->put:DB_KEYEXIST: Key/data pair already exists[E 01:28:38.529722] TROVE:DBPF:Berkeley DB: keyval_db->put:DB_KEYEXIST: Key/data pair already exists[E 10:38:11.191169] Flow proto error cleanup started on 0x2a9e04b350,error_code: -1073741973[E 10:38:11.191884] Flow proto 0x2a9e04b350 canceling a total of 1BMI or Trove operations[E 10:38:11.191910] Flow proto error cleanup finished 0x2a9e04b350,error_code: -1073741973[E 20:07:09.563212]../../../src/pvfs2/1.5.1/src/server/final-response.sm line 136:Error: PINT_encode() failure.[E 20:07:09.573061] Flow proto error cleanup started on 0x2aa347b290,error_code: -1073741973[E 20:07:09.573157] Flow proto 0x2aa347b290 canceling a total of 1BMI or Trove operations[E 20:07:09.573175] Flow proto error cleanup finished 0x2aa347b290,error_code: -1073741973[E 20:07:09.653967] [bt]/uufs/arches/sys/pkg/pvfs2/std/sbin/pvfs2-server [0x446e8b][E 20:07:09.654136] [bt]/uufs/arches/sys/pkg/pvfs2/std/sbin/pvfs2-server(main+0x890) [0x40e110][E 20:07:09.654152] [bt]/lib64/tls/libc.so.6(__libc_start_main+0xdb) [0x2a958b73fb][E 20:07:09.654165] [bt]/uufs/arches/sys/pkg/pvfs2/std/sbin/pvfs2-server [0x40c6fa][E 20:07:09.654179] Server Response 0x2aa34b0cb8 is of type:PVFS_SERV_GETCONFIG
[E 20:07:09.654194] FIXME: unimplemented resp type to print

==> ff002/var/log/pvfs2-server.log <==
[E 03:57:44.485617] tried to remove non-existant dataspace
[E 03:57:44.485707] tried to remove non-existant dataspace
[E 03:57:44.485781] tried to remove non-existant dataspace
[E 03:57:44.485870] tried to remove non-existant dataspace
[E 03:57:44.485959] tried to remove non-existant dataspace
[E 03:57:44.486058] tried to remove non-existant dataspace
[E 03:57:44.486134] tried to remove non-existant dataspace
[E 03:57:44.486224] tried to remove non-existant dataspace
[E 03:57:44.486313] tried to remove non-existant dataspace
[E 03:57:44.486390] tried to remove non-existant dataspace
[E 03:57:44.486467] tried to remove non-existant dataspace
[E 03:57:44.486554] tried to remove non-existant dataspace
[E 03:57:44.486641] tried to remove non-existant dataspace
[E 03:57:44.486740] tried to remove non-existant dataspace
[E 03:57:44.486826] tried to remove non-existant dataspace
[E 03:57:44.486913] tried to remove non-existant dataspace
[E 03:57:44.486986] tried to remove non-existant dataspace
[E 03:57:44.487074] tried to remove non-existant dataspace
[E 03:57:44.487163] tried to remove non-existant dataspace
[E 03:57:44.487251] tried to remove non-existant dataspace
[E 03:57:44.487338] tried to remove non-existant dataspace
[E 03:57:44.487436] tried to remove non-existant dataspace
[E 03:57:44.487512] tried to remove non-existant dataspace
[E 03:57:44.487611] tried to remove non-existant dataspace
[E 03:57:44.487710] tried to remove non-existant dataspace
[E 03:57:44.487784] tried to remove non-existant dataspace
[E 03:57:44.487868] tried to remove non-existant dataspace
[E 03:57:44.487954] tried to remove non-existant dataspace
[E 03:57:44.488041] tried to remove non-existant dataspace
[E 03:57:44.488139] tried to remove non-existant dataspace
[E 03:57:44.488216] tried to remove non-existant dataspace
[E 03:57:44.488301] tried to remove non-existant dataspace
[E 03:57:44.488399] tried to remove non-existant dataspace
[E 03:57:44.488475] tried to remove non-existant dataspace
[E 03:57:44.488568] tried to remove non-existant dataspace
[E 03:57:44.488655] tried to remove non-existant dataspace
[E 03:57:44.488753] tried to remove non-existant dataspace
[E 03:57:44.488840] tried to remove non-existant dataspace
[E 03:57:44.488927] tried to remove non-existant dataspace
[E 03:57:44.489001] tried to remove non-existant dataspace

==> ff003/var/log/pvfs2-server.log <==
[E 19:34:11.810075] Flow proto error cleanup finished 0x2a96915f30,error_code: -1610612737[E 19:34:11.810170] Flow proto error cleanup started on 0x2a966bd8e0,error_code: -1610612737[E 19:34:11.810485] Flow proto 0x2a966bd8e0 canceling a total of 1BMI or Trove operations[E 19:34:11.810541] Flow proto error cleanup finished 0x2a966bd8e0,error_code: -1610612737[E 19:34:11.810586] Flow proto error cleanup finished 0x2a96600660,error_code: -1610612737[E 19:34:59.872719] Job time out: cancelling flow operation, job_id:10768593.
[E 19:34:59.873205] Flow proto cancel called on 0x2a96600160
[E 19:35:15.284126] Flow proto error cleanup started on 0x2a96600160,error_code: -1610612737[E 19:35:15.286646] Flow proto 0x2a96600160 canceling a total of 2BMI or Trove operations[E 19:35:15.286691] Flow proto error cleanup finished 0x2a96600160,error_code: -1610612737[E 12:57:45.872203] Job time out: cancelling flow operation, job_id:174028738.
[E 12:57:45.872733] Flow proto cancel called on 0x2a98b56d50
[E 12:58:40.673616] Flow proto error cleanup started on 0x2a98b56d50,error_code: -1610612737[E 12:58:40.674631] Flow proto 0x2a98b56d50 canceling a total of 2BMI or Trove operations[E 12:58:40.674713] Flow proto error cleanup finished 0x2a98b56d50,error_code: -1610612737[E 11:47:31.847418] Job time out: cancelling flow operation, job_id:260685652.
[E 11:47:31.848020] Flow proto cancel called on 0x13957f0
[E 11:47:59.228124] Flow proto error cleanup started on 0x13957f0,error_code: -1610612737[E 11:47:59.228577] Flow proto 0x13957f0 canceling a total of 1 BMIor Trove operations[E 11:47:59.228600] Flow proto error cleanup finished 0x13957f0,error_code: -1610612737
[E 11:50:27.162829]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 12:25:43.563953]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 00:37:00.584569] Job time out: cancelling flow operation, job_id:60716511.
[E 00:37:00.585797] Flow proto cancel called on 0x2a96f44a90
[E 00:37:17.511475] Flow proto error cleanup started on 0x2a96f44a90,error_code: -1610612737[E 00:37:17.513090] Flow proto 0x2a96f44a90 canceling a total of 2BMI or Trove operations[E 00:37:17.513179] Flow proto error cleanup finished 0x2a96f44a90,error_code: -1610612737
[E 09:02:15.843391]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 04:44:49.759259] Flow proto error cleanup started on 0x2a9acb6ce0,error_code: -104[E 04:44:49.760934] Flow proto 0x2a9acb6ce0 canceling a total of 1BMI or Trove operations[E 04:44:49.760955] Flow proto error cleanup finished 0x2a9acb6ce0,error_code: -104[E 10:38:06.672188] Flow proto error cleanup started on 0x29b12a0,error_code: -1073741973[E 10:38:06.672815] Flow proto 0x29b12a0 canceling a total of 1 BMIor Trove operations[E 10:38:06.672841] Flow proto error cleanup finished 0x29b12a0,error_code: -1073741973[E 20:06:54.081971] Flow proto error cleanup started on 0x2aa2fc7240,error_code: -1073741973[E 20:06:54.082764] Flow proto 0x2aa2fc7240 canceling a total of 1BMI or Trove operations[E 20:06:54.082791] Flow proto error cleanup finished 0x2aa2fc7240,error_code: -1073741973
==> ff004/var/log/pvfs2-server.log <==
[E 11:40:46.991214] Flow proto cancel called on 0x2a991ae760
[E 11:40:46.991246] Flow proto error cleanup started on 0x2a991ae760,error_code: -1610612737[E 11:41:03.301929] Flow proto 0x2a991ae760 canceling a total of 2BMI or Trove operations[E 11:41:03.304713] Job time out: cancelling flow operation, job_id:260524129.
[E 11:41:03.304752] Flow proto cancel called on 0x2a994258c0
[E 11:41:03.305135] Flow proto error cleanup finished 0x2a991ae760,error_code: -1610612737[E 11:41:03.305136] Flow proto error cleanup started on 0x2a994258c0,error_code: -1610612737[E 11:41:03.305260] Flow proto 0x2a994258c0 canceling a total of 2BMI or Trove operations[E 11:41:03.305407] Flow proto error cleanup finished 0x2a994258c0,error_code: -1610612737[E 11:48:29.583000] Job time out: cancelling flow operation, job_id:260526158.
[E 11:48:29.583714] Flow proto cancel called on 0x2a98fc8680
[E 11:48:37.291564] Flow proto error cleanup started on 0x2a98fc8680,error_code: -1610612737[E 11:48:37.336903] Flow proto 0x2a98fc8680 canceling a total of 1BMI or Trove operations[E 11:48:37.366595] Flow proto error cleanup finished 0x2a98fc8680,error_code: -1610612737
[E 11:50:28.698951]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 12:25:45.480138]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 11:52:10.988318] Flow proto error cleanup started on 0x2a971436e0,error_code: -1073741973[E 11:52:10.990092] Flow proto 0x2a971436e0 canceling a total of 0BMI or Trove operations[E 11:52:10.990114] Flow proto error cleanup finished 0x2a971436e0,error_code: -1073741973[E 00:37:00.993757] Job time out: cancelling flow operation, job_id:60633518.
[E 00:37:00.994276] Flow proto cancel called on 0x2a9730ca90
[E 00:37:19.107120] Flow proto error cleanup started on 0x2a9730ca90,error_code: -1610612737[E 00:37:19.109208] Flow proto 0x2a9730ca90 canceling a total of 2BMI or Trove operations[E 00:37:19.109309] Flow proto error cleanup finished 0x2a9730ca90,error_code: -1610612737
[E 09:02:18.012373]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 04:44:49.758554] Flow proto error cleanup started on 0x2a9aac1040,error_code: -104[E 04:44:49.760161] Flow proto 0x2a9aac1040 canceling a total of 1BMI or Trove operations[E 04:44:49.760182] Flow proto error cleanup finished 0x2a9aac1040,error_code: -104[E 10:32:12.887396] Flow proto error cleanup started on 0x24788f0,error_code: -1073741973[E 10:32:12.888016] Flow proto 0x24788f0 canceling a total of 0 BMIor Trove operations[E 10:32:12.888036] Flow proto error cleanup finished 0x24788f0,error_code: -1073741973[E 10:38:16.226774] Flow proto error cleanup started on 0x2a9e01b9d0,error_code: -1073741973[E 10:38:16.227377] Flow proto 0x2a9e01b9d0 canceling a total of 1BMI or Trove operations[E 10:38:16.227401] Flow proto error cleanup finished 0x2a9e01b9d0,error_code: -1073741973[E 20:07:08.266341] Flow proto error cleanup started on 0x2aa320eec0,error_code: -1073741973[E 20:07:08.267061] Flow proto 0x2aa320eec0 canceling a total of 1BMI or Trove operations[E 20:07:08.267086] Flow proto error cleanup finished 0x2aa320eec0,error_code: -1073741973
==> ff005/var/log/pvfs2-server.log <==
[E 19:28:12.878831] Flow proto cancel called on 0x2a96620de0
[E 19:28:12.878851] Flow proto error cleanup started on 0x2a96620de0,error_code: -1610612737[E 19:28:35.501633] Flow proto 0x2a96620de0 canceling a total of 2BMI or Trove operations[E 19:28:35.502569] Flow proto error cleanup finished 0x2a96620de0,error_code: -1610612737[E 19:34:59.682130] Job time out: cancelling flow operation, job_id:10697506.
[E 19:34:59.682756] Flow proto cancel called on 0x2a965fc1c0
[E 19:35:02.829823] Flow proto error cleanup started on 0x2a965fc1c0,error_code: -1610612737[E 19:35:02.831224] Flow proto 0x2a965fc1c0 canceling a total of 2BMI or Trove operations[E 19:35:02.831323] Flow proto error cleanup finished 0x2a965fc1c0,error_code: -1610612737[E 12:57:45.167229] Job time out: cancelling flow operation, job_id:173801599.
[E 12:57:45.167954] Flow proto cancel called on 0x2a98aac730
[E 12:58:42.592382] Flow proto error cleanup started on 0x2a98aac730,error_code: -1610612737[E 12:58:42.593366] Flow proto 0x2a98aac730 canceling a total of 2BMI or Trove operations[E 12:58:42.593438] Flow proto error cleanup finished 0x2a98aac730,error_code: -1610612737
[E 11:50:30.957911]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 12:25:47.463113]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 00:37:00.895289] Job time out: cancelling flow operation, job_id:60714443.
[E 00:37:00.897106] Flow proto cancel called on 0xd0b1b0
[E 00:37:17.927882] Flow proto error cleanup started on 0xd0b1b0,error_code: -1610612737[E 00:37:17.929563] Flow proto 0xd0b1b0 canceling a total of 2 BMI orTrove operations[E 00:37:17.929641] Flow proto error cleanup finished 0xd0b1b0,error_code: -1610612737
[E 09:02:20.070652]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 04:44:49.759129] Flow proto error cleanup started on 0x2a9b1487f0,error_code: -104[E 04:44:49.760142] Flow proto 0x2a9b1487f0 canceling a total of 1BMI or Trove operations[E 04:44:49.760162] Flow proto error cleanup finished 0x2a9b1487f0,error_code: -104[E 10:32:12.894177] Flow proto error cleanup started on 0x2a9d08ad90,error_code: -1073741973[E 10:32:12.894805] Flow proto 0x2a9d08ad90 canceling a total of 0BMI or Trove operations[E 10:32:12.894823] Flow proto error cleanup finished 0x2a9d08ad90,error_code: -1073741973[E 10:38:15.483254] Flow proto error cleanup started on 0x22f0470,error_code: -1073741973[E 10:38:15.483885] Flow proto 0x22f0470 canceling a total of 1 BMIor Trove operations[E 10:38:15.483919] Flow proto error cleanup finished 0x22f0470,error_code: -1073741973[E 20:07:02.904136] Flow proto error cleanup started on 0x2aa3756870,error_code: -1073741973[E 20:07:02.904619] Flow proto 0x2aa3756870 canceling a total of 0BMI or Trove operations[E 20:07:02.904642] Flow proto error cleanup finished 0x2aa3756870,error_code: -1073741973[E 20:07:02.914124] Flow proto error cleanup started on 0x2aa3738bf0,error_code: -1073741973[E 20:07:02.914145] Flow proto 0x2aa3738bf0 canceling a total of 1BMI or Trove operations[E 20:07:02.914160] Flow proto error cleanup finished 0x2aa3738bf0,error_code: -1073741973
==> ff006/var/log/pvfs2-server.log <==
[E 19:01:47.683377]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 11:31:24.201720]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 08:34:03.161205] Flow proto error cleanup started on 0x2a971354f0,error_code: -104[E 08:34:03.162306] Flow proto 0x2a971354f0 canceling a total of 1BMI or Trove operations[E 08:34:03.162326] Flow proto error cleanup finished 0x2a971354f0,error_code: -104[E 08:36:52.935164] Job time out: cancelling flow operation, job_id:43252530.
[E 08:36:52.935787] Flow proto cancel called on 0x2a97137800
[E 08:37:00.709266] Flow proto error cleanup started on 0x2a97137800,error_code: -1610612737[E 08:37:00.710092] Flow proto 0x2a97137800 canceling a total of 2BMI or Trove operations[E 08:37:00.710171] Flow proto error cleanup finished 0x2a97137800,error_code: -1610612737[E 19:34:59.160804] Job time out: cancelling flow operation, job_id:10909203.
[E 19:34:59.161705] Flow proto cancel called on 0x2a965e5e00
[E 19:34:59.161728] Flow proto error cleanup started on 0x2a965e5e00,error_code: -1610612737[E 19:35:15.147747] Flow proto 0x2a965e5e00 canceling a total of 2BMI or Trove operations[E 19:35:15.148983] Job time out: cancelling flow operation, job_id:10909222.
[E 19:35:15.149005] Flow proto cancel called on 0x2a9622dbd0
[E 19:35:15.149214] Flow proto error cleanup finished 0x2a965e5e00,error_code: -1610612737
[E 11:50:32.274652]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 12:25:49.400128]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 00:37:00.177509] Job time out: cancelling flow operation, job_id:60674262.
[E 00:37:00.178757] Flow proto cancel called on 0x2a97285740
[E 00:37:18.801059] Flow proto error cleanup started on 0x2a97285740,error_code: -1610612737[E 00:37:18.803307] Flow proto 0x2a97285740 canceling a total of 2BMI or Trove operations[E 00:37:18.803392] Flow proto error cleanup finished 0x2a97285740,error_code: -1610612737
[E 09:02:22.157152]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 04:44:49.758640] Flow proto error cleanup started on 0x1949060,error_code: -104[E 04:44:49.759769] Flow proto 0x1949060 canceling a total of 1 BMIor Trove operations[E 04:44:49.759790] Flow proto error cleanup finished 0x1949060,error_code: -104[E 10:38:07.226797] Flow proto error cleanup started on 0x2216e60,error_code: -1073741973[E 10:38:07.227431] Flow proto 0x2216e60 canceling a total of 1 BMIor Trove operations[E 10:38:07.227485] Flow proto error cleanup finished 0x2216e60,error_code: -1073741973[E 20:07:05.613263] Flow proto error cleanup started on 0x2aa36d34c0,error_code: -1073741973[E 20:07:05.614048] Flow proto 0x2aa36d34c0 canceling a total of 1BMI or Trove operations[E 20:07:05.614069] Flow proto error cleanup finished 0x2aa36d34c0,error_code: -1073741973
==> ff007/var/log/pvfs2-server.log <==
[E 11:29:25.541128] Flow proto cancel called on 0x2a98cd0850
[E 11:30:03.976777] Flow proto error cleanup started on 0x2a98cd0850,error_code: -1610612737[E 11:30:03.978901] Flow proto 0x2a98cd0850 canceling a total of 2BMI or Trove operations[E 11:30:03.978962] Flow proto error cleanup finished 0x2a98cd0850,error_code: -1610612737[E 19:34:59.671092] Job time out: cancelling flow operation, job_id:10737005.
[E 19:34:59.672028] Flow proto cancel called on 0x2a9659c910
[E 19:35:10.519122] Flow proto error cleanup started on 0x2a9659c910,error_code: -1610612737[E 19:35:10.520406] Flow proto 0x2a9659c910 canceling a total of 2BMI or Trove operations[E 19:35:10.520536] Flow proto error cleanup finished 0x2a9659c910,error_code: -1610612737[E 12:57:45.661058] Job time out: cancelling flow operation, job_id:173821587.
[E 12:57:45.661538] Flow proto cancel called on 0x11f50f0
[E 12:58:43.551111] Flow proto error cleanup started on 0x11f50f0,error_code: -1610612737[E 12:58:43.552166] Flow proto 0x11f50f0 canceling a total of 2 BMIor Trove operations[E 12:58:43.552347] Flow proto error cleanup finished 0x11f50f0,error_code: -1610612737
[E 11:50:33.936269]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 12:25:51.724085]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 11:52:10.992619] Flow proto error cleanup started on 0x2a97091240,error_code: -1073741973[E 11:52:10.995148] Flow proto 0x2a97091240 canceling a total of 0BMI or Trove operations[E 11:52:10.995169] Flow proto error cleanup finished 0x2a97091240,error_code: -1073741973[E 00:37:00.689602] Job time out: cancelling flow operation, job_id:60688232.
[E 00:37:00.690275] Flow proto cancel called on 0x2a97604c80
[E 00:37:18.944316] Flow proto error cleanup started on 0x2a97604c80,error_code: -1610612737[E 00:37:18.946217] Flow proto 0x2a97604c80 canceling a total of 2BMI or Trove operations[E 00:37:18.946300] Flow proto error cleanup finished 0x2a97604c80,error_code: -1610612737
[E 09:02:24.269926]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 11:14:02.832222] Flow proto error cleanup started on 0x2a98040310,error_code: -104[E 11:14:02.833211] Flow proto 0x2a98040310 canceling a total of 0BMI or Trove operations[E 11:14:02.833232] Flow proto error cleanup finished 0x2a98040310,error_code: -104[E 04:44:49.757671] Flow proto error cleanup started on 0x1920720,error_code: -104[E 04:44:49.759001] Flow proto 0x1920720 canceling a total of 0 BMIor Trove operations[E 04:44:49.759020] Flow proto error cleanup finished 0x1920720,error_code: -104[E 10:38:13.450369] Flow proto error cleanup started on 0x2a9df8ff20,error_code: -1073741973[E 10:38:13.450973] Flow proto 0x2a9df8ff20 canceling a total of 1BMI or Trove operations[E 10:38:13.451011] Flow proto error cleanup finished 0x2a9df8ff20,error_code: -1073741973[E 20:06:46.818453] Flow proto error cleanup started on 0x2aa352fca0,error_code: -1073741973[E 20:06:46.819234] Flow proto 0x2aa352fca0 canceling a total of 1BMI or Trove operations[E 20:06:46.819258] Flow proto error cleanup finished 0x2aa352fca0,error_code: -1073741973
==> ff008/var/log/pvfs2-server.log <==
[E 11:23:15.053985] Job time out: cancelling flow operation, job_id:92899743.
[E 11:23:15.054655] Flow proto cancel called on 0x175fc80
[E 11:23:55.566371] Flow proto error cleanup started on 0x175fc80,error_code: -1610612737[E 11:23:55.567591] Flow proto 0x175fc80 canceling a total of 2 BMIor Trove operations[E 11:23:55.567683] Flow proto error cleanup finished 0x175fc80,error_code: -1610612737[E 11:29:25.334361] Job time out: cancelling flow operation, job_id:92911807.
[E 11:29:25.334988] Flow proto cancel called on 0x1744720
[E 11:30:02.090680] Flow proto error cleanup started on 0x1744720,error_code: -1610612737[E 11:30:02.092233] Flow proto 0x1744720 canceling a total of 2 BMIor Trove operations[E 11:30:02.092307] Flow proto error cleanup finished 0x1744720,error_code: -1610612737[E 19:34:59.158017] Job time out: cancelling flow operation, job_id:10839796.
[E 19:34:59.158967] Flow proto cancel called on 0x2a966d75d0
[E 19:35:16.526547] Flow proto error cleanup started on 0x2a966d75d0,error_code: -1610612737[E 19:35:16.528973] Flow proto 0x2a966d75d0 canceling a total of 2BMI or Trove operations[E 19:35:16.529032] Flow proto error cleanup finished 0x2a966d75d0,error_code: -1610612737[E 12:57:45.865585] Job time out: cancelling flow operation, job_id:173982292.
[E 12:57:45.866208] Flow proto cancel called on 0x11a2500
[E 12:58:41.864228] Flow proto error cleanup started on 0x11a2500,error_code: -1610612737[E 12:58:41.865154] Flow proto 0x11a2500 canceling a total of 2 BMIor Trove operations[E 12:58:41.865225] Flow proto error cleanup finished 0x11a2500,error_code: -1610612737
[E 11:50:36.038756]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 12:25:53.861118]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 00:37:00.184936] Job time out: cancelling flow operation, job_id:60608771.
[E 00:37:00.186029] Flow proto cancel called on 0xd22e60
[E 00:37:18.149785] Flow proto error cleanup started on 0xd22e60,error_code: -1610612737[E 00:37:18.151685] Flow proto 0xd22e60 canceling a total of 2 BMI orTrove operations[E 00:37:18.151774] Flow proto error cleanup finished 0xd22e60,error_code: -1610612737
[E 09:02:26.320790]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 04:44:49.757312] Flow proto error cleanup started on 0x2a9a9939e0,error_code: -104[E 04:44:49.758739] Flow proto 0x2a9a9939e0 canceling a total of 0BMI or Trove operations[E 04:44:49.758758] Flow proto error cleanup finished 0x2a9a9939e0,error_code: -104[E 10:38:03.863313] Flow proto error cleanup started on 0x2f580a0,error_code: -1073741973[E 10:38:03.863920] Flow proto 0x2f580a0 canceling a total of 1 BMIor Trove operations[E 10:38:03.863942] Flow proto error cleanup finished 0x2f580a0,error_code: -1073741973[E 20:07:12.494161] Flow proto error cleanup started on 0x2aa26f24a0,error_code: -1073741973[E 20:07:12.494667] Flow proto 0x2aa26f24a0 canceling a total of 1BMI or Trove operations[E 20:07:12.494691] Flow proto error cleanup finished 0x2aa26f24a0,error_code: -1073741973
==> ff009/var/log/pvfs2-server.log <==
[E 08:36:59.582120] Flow proto 0xd455b0 canceling a total of 2 BMI orTrove operations[E 08:36:59.582230] Flow proto error cleanup finished 0xd455b0,error_code: -1610612737[E 11:29:25.131942] Job time out: cancelling flow operation, job_id:93337739.
[E 11:29:25.132558] Flow proto cancel called on 0x2a98cf6de0
[E 11:30:04.808605] Flow proto error cleanup started on 0x2a98cf6de0,error_code: -1610612737[E 11:30:04.810303] Flow proto 0x2a98cf6de0 canceling a total of 2BMI or Trove operations[E 11:30:04.810377] Flow proto error cleanup finished 0x2a98cf6de0,error_code: -1610612737[E 19:34:59.875193] Job time out: cancelling flow operation, job_id:10673674.
[E 19:34:59.876212] Flow proto cancel called on 0x8bf5c0
[E 19:35:15.284127] Flow proto error cleanup started on 0x8bf5c0,error_code: -1610612737[E 19:35:15.287235] Flow proto 0x8bf5c0 canceling a total of 2 BMI orTrove operations[E 19:35:15.287309] Flow proto error cleanup finished 0x8bf5c0,error_code: -1610612737[E 12:57:45.762669] Job time out: cancelling flow operation, job_id:173824052.
[E 12:57:45.763290] Flow proto cancel called on 0x2a98a6b4a0
[E 12:58:43.317763] Flow proto error cleanup started on 0x2a98a6b4a0,error_code: -1610612737[E 12:58:43.318778] Flow proto 0x2a98a6b4a0 canceling a total of 2BMI or Trove operations[E 12:58:43.318851] Flow proto error cleanup finished 0x2a98a6b4a0,error_code: -1610612737
[E 11:50:38.093306]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 12:25:55.493550]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 11:52:11.051873] Flow proto error cleanup started on 0x2a971567e0,error_code: -1073741973[E 11:52:11.052839] Flow proto 0x2a971567e0 canceling a total of 0BMI or Trove operations[E 11:52:11.052858] Flow proto error cleanup finished 0x2a971567e0,error_code: -1073741973[E 00:37:00.609802] Job time out: cancelling flow operation, job_id:60604133.
[E 00:37:00.610308] Flow proto cancel called on 0x2a9734b890
[E 00:37:21.082539] Flow proto error cleanup started on 0x2a9734b890,error_code: -1610612737[E 00:37:21.084859] Flow proto 0x2a9734b890 canceling a total of 2BMI or Trove operations[E 00:37:21.084975] Flow proto error cleanup finished 0x2a9734b890,error_code: -1610612737
[E 09:02:28.364450]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 04:44:49.757475] Flow proto error cleanup started on 0x2a9abc37c0,error_code: -104[E 04:44:49.758771] Flow proto 0x2a9abc37c0 canceling a total of 1BMI or Trove operations[E 04:44:49.758791] Flow proto error cleanup finished 0x2a9abc37c0,error_code: -104[E 10:38:09.247852] Flow proto error cleanup started on 0x2a9db1e680,error_code: -1073741973[E 10:38:09.248487] Flow proto 0x2a9db1e680 canceling a total of 1BMI or Trove operations[E 10:38:09.248525] Flow proto error cleanup finished 0x2a9db1e680,error_code: -1073741973[E 20:07:03.036272] Flow proto error cleanup started on 0x2aa260b930,error_code: -1073741973[E 20:07:03.037042] Flow proto 0x2aa260b930 canceling a total of 1BMI or Trove operations[E 20:07:03.037067] Flow proto error cleanup finished 0x2aa260b930,error_code: -1073741973
==> ff010/var/log/pvfs2-server.log <==
[E 12:57:45.662629] Flow proto cancel called on 0x2a98c771f0
[E 12:58:37.177711] Flow proto error cleanup started on 0x2a98c771f0,error_code: -1610612737[E 12:58:37.178690] Flow proto 0x2a98c771f0 canceling a total of 2BMI or Trove operations[E 12:58:37.178771] Flow proto error cleanup finished 0x2a98c771f0,error_code: -1610612737[E 11:41:06.579585] Job time out: cancelling flow operation, job_id:261075524.
[E 11:41:06.580350] Flow proto cancel called on 0x12a1550
[E 11:41:17.661067] Flow proto error cleanup started on 0x12a1550,error_code: -1610612737[E 11:41:17.663997] Flow proto 0x12a1550 canceling a total of 2 BMIor Trove operations[E 11:41:17.664514] Flow proto error cleanup finished 0x12a1550,error_code: -1610612737[E 11:47:05.893222] Job time out: cancelling flow operation, job_id:261077518.
[E 11:47:05.893890] Flow proto cancel called on 0x1283ba0
[E 11:47:05.893926] Flow proto error cleanup started on 0x1283ba0,error_code: -1610612737[E 11:48:23.943123] Flow proto 0x1283ba0 canceling a total of 1 BMIor Trove operations[E 11:48:23.943696] Flow proto error cleanup finished 0x1283ba0,error_code: -1610612737
[E 11:50:40.247367]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 12:25:58.400013]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 11:52:11.007555] Flow proto error cleanup started on 0x2a96eefff0,error_code: -1073741973[E 11:52:11.008692] Flow proto 0x2a96eefff0 canceling a total of 0BMI or Trove operations[E 11:52:11.008712] Flow proto error cleanup finished 0x2a96eefff0,error_code: -1073741973[E 00:37:00.590231] Job time out: cancelling flow operation, job_id:60722692.
[E 00:37:00.590849] Flow proto cancel called on 0x2a97326e90
[E 00:37:19.214512] Flow proto error cleanup started on 0x2a97326e90,error_code: -1610612737[E 00:37:19.217037] Flow proto 0x2a97326e90 canceling a total of 2BMI or Trove operations[E 00:37:19.217121] Flow proto error cleanup finished 0x2a97326e90,error_code: -1610612737
[E 09:02:30.496085]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 11:14:02.909857] Flow proto error cleanup started on 0xd83060,error_code: -1073741973[E 11:14:02.910777] Flow proto 0xd83060 canceling a total of 0 BMI orTrove operations[E 11:14:02.910797] Flow proto error cleanup finished 0xd83060,error_code: -1073741973[E 04:44:49.757205] Flow proto error cleanup started on 0x1902ef0,error_code: -104[E 04:44:49.758324] Flow proto 0x1902ef0 canceling a total of 1 BMIor Trove operations[E 04:44:49.758344] Flow proto error cleanup finished 0x1902ef0,error_code: -104[E 10:38:12.786615] Flow proto error cleanup started on 0x2a9e17bd40,error_code: -1073741973[E 10:38:12.787231] Flow proto 0x2a9e17bd40 canceling a total of 1BMI or Trove operations[E 10:38:12.787269] Flow proto error cleanup finished 0x2a9e17bd40,error_code: -1073741973[E 20:06:58.044763] Flow proto error cleanup started on 0x2aa37d9600,error_code: -1073741973[E 20:06:58.045488] Flow proto 0x2aa37d9600 canceling a total of 1BMI or Trove operations[E 20:06:58.045512] Flow proto error cleanup finished 0x2aa37d9600,error_code: -1073741973
==> ff011/var/log/pvfs2-server.log <==
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 08:34:03.160140] Flow proto error cleanup started on 0x2a971ea2b0,error_code: -104[E 08:34:03.160818] Flow proto 0x2a971ea2b0 canceling a total of 1BMI or Trove operations[E 08:34:03.160838] Flow proto error cleanup finished 0x2a971ea2b0,error_code: -104[E 08:36:52.014916] Job time out: cancelling flow operation, job_id:43584119.
[E 08:36:52.015589] Flow proto cancel called on 0xd7e710
[E 08:36:56.733708] Flow proto error cleanup started on 0xd7e710,error_code: -1610612737[E 08:36:56.734636] Flow proto 0xd7e710 canceling a total of 2 BMI orTrove operations[E 08:36:56.734749] Flow proto error cleanup finished 0xd7e710,error_code: -1610612737[E 11:29:25.143922] Job time out: cancelling flow operation, job_id:93291033.
[E 11:29:25.144571] Flow proto cancel called on 0x2a99107a90
[E 11:30:03.333847] Flow proto error cleanup started on 0x2a99107a90,error_code: -1610612737[E 11:30:03.335664] Flow proto 0x2a99107a90 canceling a total of 2BMI or Trove operations[E 11:30:03.335726] Flow proto error cleanup finished 0x2a99107a90,error_code: -1610612737[E 12:57:45.882066] Job time out: cancelling flow operation, job_id:174098170.
[E 12:57:45.882894] Flow proto cancel called on 0x2a988d06d0
[E 12:58:44.439322] Flow proto error cleanup started on 0x2a988d06d0,error_code: -1610612737[E 12:58:44.440301] Flow proto 0x2a988d06d0 canceling a total of 2BMI or Trove operations[E 12:58:44.440381] Flow proto error cleanup finished 0x2a988d06d0,error_code: -1610612737
[E 11:50:42.315431]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 12:26:00.403797]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 11:52:11.008605] Flow proto error cleanup started on 0x2a98154cc0,error_code: -1073741973[E 11:52:11.010067] Flow proto 0x2a98154cc0 canceling a total of 0BMI or Trove operations[E 11:52:11.010087] Flow proto error cleanup finished 0x2a98154cc0,error_code: -1073741973
[E 09:02:32.419305]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 11:14:02.910124] Flow proto error cleanup started on 0x2a98297510,error_code: -1073741973[E 11:14:02.911096] Flow proto 0x2a98297510 canceling a total of 0BMI or Trove operations[E 11:14:02.911115] Flow proto error cleanup finished 0x2a98297510,error_code: -1073741973[E 04:44:49.758449] Flow proto error cleanup started on 0x2a9af08d50,error_code: -104[E 04:44:49.759700] Flow proto 0x2a9af08d50 canceling a total of 1BMI or Trove operations[E 04:44:49.759720] Flow proto error cleanup finished 0x2a9af08d50,error_code: -104[E 10:38:07.559495] Flow proto error cleanup started on 0x2a9df4d8b0,error_code: -1073741973[E 10:38:07.560125] Flow proto 0x2a9df4d8b0 canceling a total of 1BMI or Trove operations[E 10:38:07.560150] Flow proto error cleanup finished 0x2a9df4d8b0,error_code: -1073741973[E 20:07:13.662001] Flow proto error cleanup started on 0x2aa2f658f0,error_code: -1073741973[E 20:07:13.662675] Flow proto 0x2aa2f658f0 canceling a total of 1BMI or Trove operations[E 20:07:13.662703] Flow proto error cleanup finished 0x2aa2f658f0,error_code: -1073741973
==> ff012/var/log/pvfs2-server.log <==
[E 11:22:11.302587] dbpf_finalize: No such file or directory
[E 08:34:03.160263] Flow proto error cleanup started on 0x2a9738d7e0,error_code: -104[E 08:34:03.161145] Flow proto 0x2a9738d7e0 canceling a total of 1BMI or Trove operations[E 08:34:03.161165] Flow proto error cleanup finished 0x2a9738d7e0,error_code: -104[E 08:36:52.825555] Job time out: cancelling flow operation, job_id:43493526.
[E 08:36:52.826176] Flow proto cancel called on 0x2a97376500
[E 08:36:56.949573] Flow proto error cleanup started on 0x2a97376500,error_code: -1610612737[E 08:36:56.950684] Flow proto 0x2a97376500 canceling a total of 2BMI or Trove operations[E 08:36:56.950821] Flow proto error cleanup finished 0x2a97376500,error_code: -1610612737[E 11:29:25.947687] Job time out: cancelling flow operation, job_id:93095488.
[E 11:29:25.948328] Flow proto cancel called on 0x14209f0
[E 11:30:03.829131] Flow proto error cleanup started on 0x14209f0,error_code: -1610612737[E 11:30:03.830901] Flow proto 0x14209f0 canceling a total of 2 BMIor Trove operations[E 11:30:03.830976] Flow proto error cleanup finished 0x14209f0,error_code: -1610612737[E 12:57:45.976144] Job time out: cancelling flow operation, job_id:174239452.
[E 12:57:45.977003] Flow proto cancel called on 0x2a98b3d0b0
[E 12:58:37.054246] Flow proto error cleanup started on 0x2a98b3d0b0,error_code: -1610612737[E 12:58:37.055212] Flow proto 0x2a98b3d0b0 canceling a total of 2BMI or Trove operations[E 12:58:37.055292] Flow proto error cleanup finished 0x2a98b3d0b0,error_code: -1610612737
[E 11:50:44.409229]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 12:26:01.613807]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 11:52:11.015067] Flow proto error cleanup started on 0x2a971e8bd0,error_code: -1073741973[E 11:52:11.016821] Flow proto 0x2a971e8bd0 canceling a total of 0BMI or Trove operations[E 11:52:11.016847] Flow proto error cleanup finished 0x2a971e8bd0,error_code: -1073741973[E 12:41:07.917434] Flow proto error cleanup started on 0x2a9771d180,error_code: -1073741973[E 12:41:07.917915] Flow proto 0x2a9771d180 canceling a total of 0BMI or Trove operations[E 12:41:07.917934] Flow proto error cleanup finished 0x2a9771d180,error_code: -1073741973
[E 09:02:34.417028]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 04:44:49.758474] Flow proto error cleanup started on 0x2a9ad31b10,error_code: -104[E 04:44:49.759557] Flow proto 0x2a9ad31b10 canceling a total of 1BMI or Trove operations[E 04:44:49.759578] Flow proto error cleanup finished 0x2a9ad31b10,error_code: -104[E 10:38:11.060375] Flow proto error cleanup started on 0x2a9dd0a2c0,error_code: -1073741973[E 10:38:11.061020] Flow proto 0x2a9dd0a2c0 canceling a total of 1BMI or Trove operations[E 10:38:11.061044] Flow proto error cleanup finished 0x2a9dd0a2c0,error_code: -1073741973[E 20:07:04.830849] Flow proto error cleanup started on 0x2aa2e753f0,error_code: -1073741973[E 20:07:04.831496] Flow proto 0x2aa2e753f0 canceling a total of 1BMI or Trove operations[E 20:07:04.831523] Flow proto error cleanup finished 0x2aa2e753f0,error_code: -1073741973
------------------------------------------------------------------------

Subject:
Molpro details
From:
"Anita Orendt" <[EMAIL PROTECTED]>
Date:
Tue, 10 Oct 2006 12:17:27 -0600
To:
"Brian Haymore" <[EMAIL PROTECTED]>

To:
"Brian Haymore" <[EMAIL PROTECTED]>
Molpro is a quantum package detailed at www.molpro.netI am working with a serial installation of 2006.1 version.
With either /scratch/serial or /tmp all 271 tests pass.
With /scratch/parallel all fail right at the start (259 failed, not sure
what is going on with remaining 12 as they list as successful but there
is no output) with:


Write error in iow_direct_write; fd=9, l=4, p=4096; write returns -1
This may indicate a filled disk, or that a disk quota has been exceeded

 ERROR WRITING            4 WORDS AT OFFSET       4096. TO FILE 1
IMPLEMENTATION=df   FILE HANDLE=     9  IERR=   -32
 ? Error
 ? I/O error
 ? The problem occurs in writew

 ERROR EXIT
 CURRENT STACK:      MAIN
Anita
********************************
Anita M. Orendt
Staff Scientist for Molecular Sciences
Center for High Performance Computing
University of Utah
155 South 1452 East Room 422
Salt Lake City, UT 84112-0190
email: [EMAIL PROTECTED]
phone: 801-231-2762
fax: 801-585-5366
*********************************
------------------------------------------------------------------------
_______________________________________________
Pvfs2-developers mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers


_______________________________________________
Pvfs2-developers mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers

Re: [Pvfs2-developers] [Fwd: Molpro details]

Reply via email to