[Pvfs2-developers] [Fwd: Molpro details]

Tue, 10 Oct 2006 11:59:53 -0700

We are seeing another application having issues accessing pvfs2 space via the client side mount via the kernel module. I'm forwarding a description of things and looking for help on what info to gather to add to this report. What I have so far is the client and server log output from during her runs of these test cases. 

Client syslog messages:

Oct 10 12:40:58 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017704, -- returning -2
Oct 10 12:40:59 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017704, -- returning -2
Oct 10 12:40:59 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017703, -- returning -2
Oct 10 12:40:59 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017703, -- returning -2
Oct 10 12:40:59 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017702, -- returning -2
Oct 10 12:41:00 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017702, -- returning -2
Oct 10 12:41:00 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017700, -- returning -2
Oct 10 12:41:02 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017698, -- returning -2
Oct 10 12:41:04 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017697, -- returning -2
Oct 10 12:41:06 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017697, -- returning -2
Oct 10 12:41:06 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017696, -- returning -2
Oct 10 12:41:07 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017695, -- returning -2
Oct 10 12:41:07 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017695, -- returning -2
Oct 10 12:41:07 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017694, -- returning -2
Oct 10 12:41:08 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017694, -- returning -2
Oct 10 12:41:08 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017693, -- returning -2
Oct 10 12:41:09 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017693, -- returning -2
Oct 10 12:41:09 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017692, -- returning -2
Oct 10 12:41:09 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017692, -- returning -2
Oct 10 12:41:09 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017691, -- returning -2
Oct 10 12:41:09 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017691, -- returning -2
Oct 10 12:41:09 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017690, -- returning -2
Oct 10 12:41:10 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017689, -- returning -2
Oct 10 12:41:11 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017689, -- returning -2
Oct 10 12:41:11 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017688, -- returning -2
Oct 10 12:41:11 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017687, -- returning -2
Oct 10 12:41:12 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017686, -- returning -2
Oct 10 12:41:12 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017685, -- returning -2
Oct 10 12:41:13 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017684, -- returning -2
Oct 10 12:41:13 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017684, -- returning -2
Oct 10 12:41:13 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017683, -- returning -2
Oct 10 12:41:14 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017683, -- returning -2
Oct 10 12:41:14 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017682, -- returning -2
Oct 10 12:41:14 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017681, -- returning -2
Oct 10 12:41:15 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017680, -- returning -2
Oct 10 12:41:16 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017680, -- returning -2
Oct 10 12:41:16 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017679, -- returning -2
Oct 10 12:41:17 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017679, -- returning -2
Oct 10 12:41:17 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017678, -- returning -2
Oct 10 12:41:18 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017678, -- returning -2
Oct 10 12:41:18 la099.landscapearch.arches kernel: pvfs2_file_write: error 
writing to handle 1017677, -- returning -2



Server pvfs2 logs showing last 40 lines from each of the 12 servers:

[EMAIL PROTECTED] snapshot]# tail -40 ff*/var/log/pvfs*
==> ff001/var/log/pvfs2-server.log <==
[E 01:28:36.025839] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: 
Key/data pair already exists
[E 01:28:36.039854] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: 
Key/data pair already exists
[E 01:28:36.042510] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: 
Key/data pair already exists
[E 01:28:36.269407] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: 
Key/data pair already exists
[E 01:28:36.269550] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: 
Key/data pair already exists
[E 01:28:36.269672] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: 
Key/data pair already exists
[E 01:28:36.565606] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: 
Key/data pair already exists
[E 01:28:36.565808] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: 
Key/data pair already exists
[E 01:28:36.566065] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: 
Key/data pair already exists
[E 01:28:37.320512] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: 
Key/data pair already exists
[E 01:28:37.391505] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: 
Key/data pair already exists
[E 01:28:37.391707] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: 
Key/data pair already exists
[E 01:28:37.605731] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: 
Key/data pair already exists
[E 01:28:37.616642] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: 
Key/data pair already exists
[E 01:28:37.643355] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: 
Key/data pair already exists
[E 01:28:37.830465] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: 
Key/data pair already exists
[E 01:28:37.831224] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: 
Key/data pair already exists
[E 01:28:37.831336] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: 
Key/data pair already exists
[E 01:28:37.925518] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: 
Key/data pair already exists
[E 01:28:37.925677] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: 
Key/data pair already exists
[E 01:28:37.925791] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: 
Key/data pair already exists
[E 01:28:38.268132] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: 
Key/data pair already exists
[E 01:28:38.268692] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: 
Key/data pair already exists
[E 01:28:38.291768] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: 
Key/data pair already exists
[E 01:28:38.511212] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: 
Key/data pair already exists
[E 01:28:38.529310] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: 
Key/data pair already exists
[E 01:28:38.529722] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: 
Key/data pair already exists
[E 10:38:11.191169] Flow proto error cleanup started on 0x2a9e04b350, 
error_code: -1073741973
[E 10:38:11.191884] Flow proto 0x2a9e04b350 canceling a total of 1 BMI or Trove 
operations
[E 10:38:11.191910] Flow proto error cleanup finished 0x2a9e04b350, error_code: 
-1073741973
[E 20:07:09.563212] ../../../src/pvfs2/1.5.1/src/server/final-response.sm line 
136: Error: PINT_encode() failure.
[E 20:07:09.573061] Flow proto error cleanup started on 0x2aa347b290, 
error_code: -1073741973
[E 20:07:09.573157] Flow proto 0x2aa347b290 canceling a total of 1 BMI or Trove 
operations
[E 20:07:09.573175] Flow proto error cleanup finished 0x2aa347b290, error_code: 
-1073741973
[E 20:07:09.653967]     [bt] /uufs/arches/sys/pkg/pvfs2/std/sbin/pvfs2-server 
[0x446e8b]
[E 20:07:09.654136]     [bt] 
/uufs/arches/sys/pkg/pvfs2/std/sbin/pvfs2-server(main+0x890) [0x40e110]
[E 20:07:09.654152]     [bt] /lib64/tls/libc.so.6(__libc_start_main+0xdb) 
[0x2a958b73fb]
[E 20:07:09.654165]     [bt] /uufs/arches/sys/pkg/pvfs2/std/sbin/pvfs2-server 
[0x40c6fa]
[E 20:07:09.654179] Server Response 0x2aa34b0cb8 is of type: PVFS_SERV_GETCONFIG
[E 20:07:09.654194] FIXME: unimplemented resp type to print

==> ff002/var/log/pvfs2-server.log <==
[E 03:57:44.485617] tried to remove non-existant dataspace
[E 03:57:44.485707] tried to remove non-existant dataspace
[E 03:57:44.485781] tried to remove non-existant dataspace
[E 03:57:44.485870] tried to remove non-existant dataspace
[E 03:57:44.485959] tried to remove non-existant dataspace
[E 03:57:44.486058] tried to remove non-existant dataspace
[E 03:57:44.486134] tried to remove non-existant dataspace
[E 03:57:44.486224] tried to remove non-existant dataspace
[E 03:57:44.486313] tried to remove non-existant dataspace
[E 03:57:44.486390] tried to remove non-existant dataspace
[E 03:57:44.486467] tried to remove non-existant dataspace
[E 03:57:44.486554] tried to remove non-existant dataspace
[E 03:57:44.486641] tried to remove non-existant dataspace
[E 03:57:44.486740] tried to remove non-existant dataspace
[E 03:57:44.486826] tried to remove non-existant dataspace
[E 03:57:44.486913] tried to remove non-existant dataspace
[E 03:57:44.486986] tried to remove non-existant dataspace
[E 03:57:44.487074] tried to remove non-existant dataspace
[E 03:57:44.487163] tried to remove non-existant dataspace
[E 03:57:44.487251] tried to remove non-existant dataspace
[E 03:57:44.487338] tried to remove non-existant dataspace
[E 03:57:44.487436] tried to remove non-existant dataspace
[E 03:57:44.487512] tried to remove non-existant dataspace
[E 03:57:44.487611] tried to remove non-existant dataspace
[E 03:57:44.487710] tried to remove non-existant dataspace
[E 03:57:44.487784] tried to remove non-existant dataspace
[E 03:57:44.487868] tried to remove non-existant dataspace
[E 03:57:44.487954] tried to remove non-existant dataspace
[E 03:57:44.488041] tried to remove non-existant dataspace
[E 03:57:44.488139] tried to remove non-existant dataspace
[E 03:57:44.488216] tried to remove non-existant dataspace
[E 03:57:44.488301] tried to remove non-existant dataspace
[E 03:57:44.488399] tried to remove non-existant dataspace
[E 03:57:44.488475] tried to remove non-existant dataspace
[E 03:57:44.488568] tried to remove non-existant dataspace
[E 03:57:44.488655] tried to remove non-existant dataspace
[E 03:57:44.488753] tried to remove non-existant dataspace
[E 03:57:44.488840] tried to remove non-existant dataspace
[E 03:57:44.488927] tried to remove non-existant dataspace
[E 03:57:44.489001] tried to remove non-existant dataspace

==> ff003/var/log/pvfs2-server.log <==
[E 19:34:11.810075] Flow proto error cleanup finished 0x2a96915f30, error_code: 
-1610612737
[E 19:34:11.810170] Flow proto error cleanup started on 0x2a966bd8e0, 
error_code: -1610612737
[E 19:34:11.810485] Flow proto 0x2a966bd8e0 canceling a total of 1 BMI or Trove 
operations
[E 19:34:11.810541] Flow proto error cleanup finished 0x2a966bd8e0, error_code: 
-1610612737
[E 19:34:11.810586] Flow proto error cleanup finished 0x2a96600660, error_code: 
-1610612737
[E 19:34:59.872719] Job time out: cancelling flow operation, job_id: 10768593.
[E 19:34:59.873205] Flow proto cancel called on 0x2a96600160
[E 19:35:15.284126] Flow proto error cleanup started on 0x2a96600160, 
error_code: -1610612737
[E 19:35:15.286646] Flow proto 0x2a96600160 canceling a total of 2 BMI or Trove 
operations
[E 19:35:15.286691] Flow proto error cleanup finished 0x2a96600160, error_code: 
-1610612737
[E 12:57:45.872203] Job time out: cancelling flow operation, job_id: 174028738.
[E 12:57:45.872733] Flow proto cancel called on 0x2a98b56d50
[E 12:58:40.673616] Flow proto error cleanup started on 0x2a98b56d50, 
error_code: -1610612737
[E 12:58:40.674631] Flow proto 0x2a98b56d50 canceling a total of 2 BMI or Trove 
operations
[E 12:58:40.674713] Flow proto error cleanup finished 0x2a98b56d50, error_code: 
-1610612737
[E 11:47:31.847418] Job time out: cancelling flow operation, job_id: 260685652.
[E 11:47:31.848020] Flow proto cancel called on 0x13957f0
[E 11:47:59.228124] Flow proto error cleanup started on 0x13957f0, error_code: 
-1610612737
[E 11:47:59.228577] Flow proto 0x13957f0 canceling a total of 1 BMI or Trove 
operations
[E 11:47:59.228600] Flow proto error cleanup finished 0x13957f0, error_code: 
-1610612737
[E 11:50:27.162829]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 12:25:43.563953]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 00:37:00.584569] Job time out: cancelling flow operation, job_id: 60716511.
[E 00:37:00.585797] Flow proto cancel called on 0x2a96f44a90
[E 00:37:17.511475] Flow proto error cleanup started on 0x2a96f44a90, 
error_code: -1610612737
[E 00:37:17.513090] Flow proto 0x2a96f44a90 canceling a total of 2 BMI or Trove 
operations
[E 00:37:17.513179] Flow proto error cleanup finished 0x2a96f44a90, error_code: 
-1610612737
[E 09:02:15.843391]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 04:44:49.759259] Flow proto error cleanup started on 0x2a9acb6ce0, 
error_code: -104
[E 04:44:49.760934] Flow proto 0x2a9acb6ce0 canceling a total of 1 BMI or Trove 
operations
[E 04:44:49.760955] Flow proto error cleanup finished 0x2a9acb6ce0, error_code: 
-104
[E 10:38:06.672188] Flow proto error cleanup started on 0x29b12a0, error_code: 
-1073741973
[E 10:38:06.672815] Flow proto 0x29b12a0 canceling a total of 1 BMI or Trove 
operations
[E 10:38:06.672841] Flow proto error cleanup finished 0x29b12a0, error_code: 
-1073741973
[E 20:06:54.081971] Flow proto error cleanup started on 0x2aa2fc7240, 
error_code: -1073741973
[E 20:06:54.082764] Flow proto 0x2aa2fc7240 canceling a total of 1 BMI or Trove 
operations
[E 20:06:54.082791] Flow proto error cleanup finished 0x2aa2fc7240, error_code: 
-1073741973

==> ff004/var/log/pvfs2-server.log <==
[E 11:40:46.991214] Flow proto cancel called on 0x2a991ae760
[E 11:40:46.991246] Flow proto error cleanup started on 0x2a991ae760, 
error_code: -1610612737
[E 11:41:03.301929] Flow proto 0x2a991ae760 canceling a total of 2 BMI or Trove 
operations
[E 11:41:03.304713] Job time out: cancelling flow operation, job_id: 260524129.
[E 11:41:03.304752] Flow proto cancel called on 0x2a994258c0
[E 11:41:03.305135] Flow proto error cleanup finished 0x2a991ae760, error_code: 
-1610612737
[E 11:41:03.305136] Flow proto error cleanup started on 0x2a994258c0, 
error_code: -1610612737
[E 11:41:03.305260] Flow proto 0x2a994258c0 canceling a total of 2 BMI or Trove 
operations
[E 11:41:03.305407] Flow proto error cleanup finished 0x2a994258c0, error_code: 
-1610612737
[E 11:48:29.583000] Job time out: cancelling flow operation, job_id: 260526158.
[E 11:48:29.583714] Flow proto cancel called on 0x2a98fc8680
[E 11:48:37.291564] Flow proto error cleanup started on 0x2a98fc8680, 
error_code: -1610612737
[E 11:48:37.336903] Flow proto 0x2a98fc8680 canceling a total of 1 BMI or Trove 
operations
[E 11:48:37.366595] Flow proto error cleanup finished 0x2a98fc8680, error_code: 
-1610612737
[E 11:50:28.698951]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 12:25:45.480138]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 11:52:10.988318] Flow proto error cleanup started on 0x2a971436e0, 
error_code: -1073741973
[E 11:52:10.990092] Flow proto 0x2a971436e0 canceling a total of 0 BMI or Trove 
operations
[E 11:52:10.990114] Flow proto error cleanup finished 0x2a971436e0, error_code: 
-1073741973
[E 00:37:00.993757] Job time out: cancelling flow operation, job_id: 60633518.
[E 00:37:00.994276] Flow proto cancel called on 0x2a9730ca90
[E 00:37:19.107120] Flow proto error cleanup started on 0x2a9730ca90, 
error_code: -1610612737
[E 00:37:19.109208] Flow proto 0x2a9730ca90 canceling a total of 2 BMI or Trove 
operations
[E 00:37:19.109309] Flow proto error cleanup finished 0x2a9730ca90, error_code: 
-1610612737
[E 09:02:18.012373]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 04:44:49.758554] Flow proto error cleanup started on 0x2a9aac1040, 
error_code: -104
[E 04:44:49.760161] Flow proto 0x2a9aac1040 canceling a total of 1 BMI or Trove 
operations
[E 04:44:49.760182] Flow proto error cleanup finished 0x2a9aac1040, error_code: 
-104
[E 10:32:12.887396] Flow proto error cleanup started on 0x24788f0, error_code: 
-1073741973
[E 10:32:12.888016] Flow proto 0x24788f0 canceling a total of 0 BMI or Trove 
operations
[E 10:32:12.888036] Flow proto error cleanup finished 0x24788f0, error_code: 
-1073741973
[E 10:38:16.226774] Flow proto error cleanup started on 0x2a9e01b9d0, 
error_code: -1073741973
[E 10:38:16.227377] Flow proto 0x2a9e01b9d0 canceling a total of 1 BMI or Trove 
operations
[E 10:38:16.227401] Flow proto error cleanup finished 0x2a9e01b9d0, error_code: 
-1073741973
[E 20:07:08.266341] Flow proto error cleanup started on 0x2aa320eec0, 
error_code: -1073741973
[E 20:07:08.267061] Flow proto 0x2aa320eec0 canceling a total of 1 BMI or Trove 
operations
[E 20:07:08.267086] Flow proto error cleanup finished 0x2aa320eec0, error_code: 
-1073741973

==> ff005/var/log/pvfs2-server.log <==
[E 19:28:12.878831] Flow proto cancel called on 0x2a96620de0
[E 19:28:12.878851] Flow proto error cleanup started on 0x2a96620de0, 
error_code: -1610612737
[E 19:28:35.501633] Flow proto 0x2a96620de0 canceling a total of 2 BMI or Trove 
operations
[E 19:28:35.502569] Flow proto error cleanup finished 0x2a96620de0, error_code: 
-1610612737
[E 19:34:59.682130] Job time out: cancelling flow operation, job_id: 10697506.
[E 19:34:59.682756] Flow proto cancel called on 0x2a965fc1c0
[E 19:35:02.829823] Flow proto error cleanup started on 0x2a965fc1c0, 
error_code: -1610612737
[E 19:35:02.831224] Flow proto 0x2a965fc1c0 canceling a total of 2 BMI or Trove 
operations
[E 19:35:02.831323] Flow proto error cleanup finished 0x2a965fc1c0, error_code: 
-1610612737
[E 12:57:45.167229] Job time out: cancelling flow operation, job_id: 173801599.
[E 12:57:45.167954] Flow proto cancel called on 0x2a98aac730
[E 12:58:42.592382] Flow proto error cleanup started on 0x2a98aac730, 
error_code: -1610612737
[E 12:58:42.593366] Flow proto 0x2a98aac730 canceling a total of 2 BMI or Trove 
operations
[E 12:58:42.593438] Flow proto error cleanup finished 0x2a98aac730, error_code: 
-1610612737
[E 11:50:30.957911]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 12:25:47.463113]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 00:37:00.895289] Job time out: cancelling flow operation, job_id: 60714443.
[E 00:37:00.897106] Flow proto cancel called on 0xd0b1b0
[E 00:37:17.927882] Flow proto error cleanup started on 0xd0b1b0, error_code: 
-1610612737
[E 00:37:17.929563] Flow proto 0xd0b1b0 canceling a total of 2 BMI or Trove 
operations
[E 00:37:17.929641] Flow proto error cleanup finished 0xd0b1b0, error_code: 
-1610612737
[E 09:02:20.070652]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 04:44:49.759129] Flow proto error cleanup started on 0x2a9b1487f0, 
error_code: -104
[E 04:44:49.760142] Flow proto 0x2a9b1487f0 canceling a total of 1 BMI or Trove 
operations
[E 04:44:49.760162] Flow proto error cleanup finished 0x2a9b1487f0, error_code: 
-104
[E 10:32:12.894177] Flow proto error cleanup started on 0x2a9d08ad90, 
error_code: -1073741973
[E 10:32:12.894805] Flow proto 0x2a9d08ad90 canceling a total of 0 BMI or Trove 
operations
[E 10:32:12.894823] Flow proto error cleanup finished 0x2a9d08ad90, error_code: 
-1073741973
[E 10:38:15.483254] Flow proto error cleanup started on 0x22f0470, error_code: 
-1073741973
[E 10:38:15.483885] Flow proto 0x22f0470 canceling a total of 1 BMI or Trove 
operations
[E 10:38:15.483919] Flow proto error cleanup finished 0x22f0470, error_code: 
-1073741973
[E 20:07:02.904136] Flow proto error cleanup started on 0x2aa3756870, 
error_code: -1073741973
[E 20:07:02.904619] Flow proto 0x2aa3756870 canceling a total of 0 BMI or Trove 
operations
[E 20:07:02.904642] Flow proto error cleanup finished 0x2aa3756870, error_code: 
-1073741973
[E 20:07:02.914124] Flow proto error cleanup started on 0x2aa3738bf0, 
error_code: -1073741973
[E 20:07:02.914145] Flow proto 0x2aa3738bf0 canceling a total of 1 BMI or Trove 
operations
[E 20:07:02.914160] Flow proto error cleanup finished 0x2aa3738bf0, error_code: 
-1073741973

==> ff006/var/log/pvfs2-server.log <==
[E 19:01:47.683377]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 11:31:24.201720]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 08:34:03.161205] Flow proto error cleanup started on 0x2a971354f0, 
error_code: -104
[E 08:34:03.162306] Flow proto 0x2a971354f0 canceling a total of 1 BMI or Trove 
operations
[E 08:34:03.162326] Flow proto error cleanup finished 0x2a971354f0, error_code: 
-104
[E 08:36:52.935164] Job time out: cancelling flow operation, job_id: 43252530.
[E 08:36:52.935787] Flow proto cancel called on 0x2a97137800
[E 08:37:00.709266] Flow proto error cleanup started on 0x2a97137800, 
error_code: -1610612737
[E 08:37:00.710092] Flow proto 0x2a97137800 canceling a total of 2 BMI or Trove 
operations
[E 08:37:00.710171] Flow proto error cleanup finished 0x2a97137800, error_code: 
-1610612737
[E 19:34:59.160804] Job time out: cancelling flow operation, job_id: 10909203.
[E 19:34:59.161705] Flow proto cancel called on 0x2a965e5e00
[E 19:34:59.161728] Flow proto error cleanup started on 0x2a965e5e00, 
error_code: -1610612737
[E 19:35:15.147747] Flow proto 0x2a965e5e00 canceling a total of 2 BMI or Trove 
operations
[E 19:35:15.148983] Job time out: cancelling flow operation, job_id: 10909222.
[E 19:35:15.149005] Flow proto cancel called on 0x2a9622dbd0
[E 19:35:15.149214] Flow proto error cleanup finished 0x2a965e5e00, error_code: 
-1610612737
[E 11:50:32.274652]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 12:25:49.400128]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 00:37:00.177509] Job time out: cancelling flow operation, job_id: 60674262.
[E 00:37:00.178757] Flow proto cancel called on 0x2a97285740
[E 00:37:18.801059] Flow proto error cleanup started on 0x2a97285740, 
error_code: -1610612737
[E 00:37:18.803307] Flow proto 0x2a97285740 canceling a total of 2 BMI or Trove 
operations
[E 00:37:18.803392] Flow proto error cleanup finished 0x2a97285740, error_code: 
-1610612737
[E 09:02:22.157152]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 04:44:49.758640] Flow proto error cleanup started on 0x1949060, error_code: 
-104
[E 04:44:49.759769] Flow proto 0x1949060 canceling a total of 1 BMI or Trove 
operations
[E 04:44:49.759790] Flow proto error cleanup finished 0x1949060, error_code: 
-104
[E 10:38:07.226797] Flow proto error cleanup started on 0x2216e60, error_code: 
-1073741973
[E 10:38:07.227431] Flow proto 0x2216e60 canceling a total of 1 BMI or Trove 
operations
[E 10:38:07.227485] Flow proto error cleanup finished 0x2216e60, error_code: 
-1073741973
[E 20:07:05.613263] Flow proto error cleanup started on 0x2aa36d34c0, 
error_code: -1073741973
[E 20:07:05.614048] Flow proto 0x2aa36d34c0 canceling a total of 1 BMI or Trove 
operations
[E 20:07:05.614069] Flow proto error cleanup finished 0x2aa36d34c0, error_code: 
-1073741973

==> ff007/var/log/pvfs2-server.log <==
[E 11:29:25.541128] Flow proto cancel called on 0x2a98cd0850
[E 11:30:03.976777] Flow proto error cleanup started on 0x2a98cd0850, 
error_code: -1610612737
[E 11:30:03.978901] Flow proto 0x2a98cd0850 canceling a total of 2 BMI or Trove 
operations
[E 11:30:03.978962] Flow proto error cleanup finished 0x2a98cd0850, error_code: 
-1610612737
[E 19:34:59.671092] Job time out: cancelling flow operation, job_id: 10737005.
[E 19:34:59.672028] Flow proto cancel called on 0x2a9659c910
[E 19:35:10.519122] Flow proto error cleanup started on 0x2a9659c910, 
error_code: -1610612737
[E 19:35:10.520406] Flow proto 0x2a9659c910 canceling a total of 2 BMI or Trove 
operations
[E 19:35:10.520536] Flow proto error cleanup finished 0x2a9659c910, error_code: 
-1610612737
[E 12:57:45.661058] Job time out: cancelling flow operation, job_id: 173821587.
[E 12:57:45.661538] Flow proto cancel called on 0x11f50f0
[E 12:58:43.551111] Flow proto error cleanup started on 0x11f50f0, error_code: 
-1610612737
[E 12:58:43.552166] Flow proto 0x11f50f0 canceling a total of 2 BMI or Trove 
operations
[E 12:58:43.552347] Flow proto error cleanup finished 0x11f50f0, error_code: 
-1610612737
[E 11:50:33.936269]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 12:25:51.724085]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 11:52:10.992619] Flow proto error cleanup started on 0x2a97091240, 
error_code: -1073741973
[E 11:52:10.995148] Flow proto 0x2a97091240 canceling a total of 0 BMI or Trove 
operations
[E 11:52:10.995169] Flow proto error cleanup finished 0x2a97091240, error_code: 
-1073741973
[E 00:37:00.689602] Job time out: cancelling flow operation, job_id: 60688232.
[E 00:37:00.690275] Flow proto cancel called on 0x2a97604c80
[E 00:37:18.944316] Flow proto error cleanup started on 0x2a97604c80, 
error_code: -1610612737
[E 00:37:18.946217] Flow proto 0x2a97604c80 canceling a total of 2 BMI or Trove 
operations
[E 00:37:18.946300] Flow proto error cleanup finished 0x2a97604c80, error_code: 
-1610612737
[E 09:02:24.269926]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 11:14:02.832222] Flow proto error cleanup started on 0x2a98040310, 
error_code: -104
[E 11:14:02.833211] Flow proto 0x2a98040310 canceling a total of 0 BMI or Trove 
operations
[E 11:14:02.833232] Flow proto error cleanup finished 0x2a98040310, error_code: 
-104
[E 04:44:49.757671] Flow proto error cleanup started on 0x1920720, error_code: 
-104
[E 04:44:49.759001] Flow proto 0x1920720 canceling a total of 0 BMI or Trove 
operations
[E 04:44:49.759020] Flow proto error cleanup finished 0x1920720, error_code: 
-104
[E 10:38:13.450369] Flow proto error cleanup started on 0x2a9df8ff20, 
error_code: -1073741973
[E 10:38:13.450973] Flow proto 0x2a9df8ff20 canceling a total of 1 BMI or Trove 
operations
[E 10:38:13.451011] Flow proto error cleanup finished 0x2a9df8ff20, error_code: 
-1073741973
[E 20:06:46.818453] Flow proto error cleanup started on 0x2aa352fca0, 
error_code: -1073741973
[E 20:06:46.819234] Flow proto 0x2aa352fca0 canceling a total of 1 BMI or Trove 
operations
[E 20:06:46.819258] Flow proto error cleanup finished 0x2aa352fca0, error_code: 
-1073741973

==> ff008/var/log/pvfs2-server.log <==
[E 11:23:15.053985] Job time out: cancelling flow operation, job_id: 92899743.
[E 11:23:15.054655] Flow proto cancel called on 0x175fc80
[E 11:23:55.566371] Flow proto error cleanup started on 0x175fc80, error_code: 
-1610612737
[E 11:23:55.567591] Flow proto 0x175fc80 canceling a total of 2 BMI or Trove 
operations
[E 11:23:55.567683] Flow proto error cleanup finished 0x175fc80, error_code: 
-1610612737
[E 11:29:25.334361] Job time out: cancelling flow operation, job_id: 92911807.
[E 11:29:25.334988] Flow proto cancel called on 0x1744720
[E 11:30:02.090680] Flow proto error cleanup started on 0x1744720, error_code: 
-1610612737
[E 11:30:02.092233] Flow proto 0x1744720 canceling a total of 2 BMI or Trove 
operations
[E 11:30:02.092307] Flow proto error cleanup finished 0x1744720, error_code: 
-1610612737
[E 19:34:59.158017] Job time out: cancelling flow operation, job_id: 10839796.
[E 19:34:59.158967] Flow proto cancel called on 0x2a966d75d0
[E 19:35:16.526547] Flow proto error cleanup started on 0x2a966d75d0, 
error_code: -1610612737
[E 19:35:16.528973] Flow proto 0x2a966d75d0 canceling a total of 2 BMI or Trove 
operations
[E 19:35:16.529032] Flow proto error cleanup finished 0x2a966d75d0, error_code: 
-1610612737
[E 12:57:45.865585] Job time out: cancelling flow operation, job_id: 173982292.
[E 12:57:45.866208] Flow proto cancel called on 0x11a2500
[E 12:58:41.864228] Flow proto error cleanup started on 0x11a2500, error_code: 
-1610612737
[E 12:58:41.865154] Flow proto 0x11a2500 canceling a total of 2 BMI or Trove 
operations
[E 12:58:41.865225] Flow proto error cleanup finished 0x11a2500, error_code: 
-1610612737
[E 11:50:36.038756]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 12:25:53.861118]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 00:37:00.184936] Job time out: cancelling flow operation, job_id: 60608771.
[E 00:37:00.186029] Flow proto cancel called on 0xd22e60
[E 00:37:18.149785] Flow proto error cleanup started on 0xd22e60, error_code: 
-1610612737
[E 00:37:18.151685] Flow proto 0xd22e60 canceling a total of 2 BMI or Trove 
operations
[E 00:37:18.151774] Flow proto error cleanup finished 0xd22e60, error_code: 
-1610612737
[E 09:02:26.320790]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 04:44:49.757312] Flow proto error cleanup started on 0x2a9a9939e0, 
error_code: -104
[E 04:44:49.758739] Flow proto 0x2a9a9939e0 canceling a total of 0 BMI or Trove 
operations
[E 04:44:49.758758] Flow proto error cleanup finished 0x2a9a9939e0, error_code: 
-104
[E 10:38:03.863313] Flow proto error cleanup started on 0x2f580a0, error_code: 
-1073741973
[E 10:38:03.863920] Flow proto 0x2f580a0 canceling a total of 1 BMI or Trove 
operations
[E 10:38:03.863942] Flow proto error cleanup finished 0x2f580a0, error_code: 
-1073741973
[E 20:07:12.494161] Flow proto error cleanup started on 0x2aa26f24a0, 
error_code: -1073741973
[E 20:07:12.494667] Flow proto 0x2aa26f24a0 canceling a total of 1 BMI or Trove 
operations
[E 20:07:12.494691] Flow proto error cleanup finished 0x2aa26f24a0, error_code: 
-1073741973

==> ff009/var/log/pvfs2-server.log <==
[E 08:36:59.582120] Flow proto 0xd455b0 canceling a total of 2 BMI or Trove 
operations
[E 08:36:59.582230] Flow proto error cleanup finished 0xd455b0, error_code: 
-1610612737
[E 11:29:25.131942] Job time out: cancelling flow operation, job_id: 93337739.
[E 11:29:25.132558] Flow proto cancel called on 0x2a98cf6de0
[E 11:30:04.808605] Flow proto error cleanup started on 0x2a98cf6de0, 
error_code: -1610612737
[E 11:30:04.810303] Flow proto 0x2a98cf6de0 canceling a total of 2 BMI or Trove 
operations
[E 11:30:04.810377] Flow proto error cleanup finished 0x2a98cf6de0, error_code: 
-1610612737
[E 19:34:59.875193] Job time out: cancelling flow operation, job_id: 10673674.
[E 19:34:59.876212] Flow proto cancel called on 0x8bf5c0
[E 19:35:15.284127] Flow proto error cleanup started on 0x8bf5c0, error_code: 
-1610612737
[E 19:35:15.287235] Flow proto 0x8bf5c0 canceling a total of 2 BMI or Trove 
operations
[E 19:35:15.287309] Flow proto error cleanup finished 0x8bf5c0, error_code: 
-1610612737
[E 12:57:45.762669] Job time out: cancelling flow operation, job_id: 173824052.
[E 12:57:45.763290] Flow proto cancel called on 0x2a98a6b4a0
[E 12:58:43.317763] Flow proto error cleanup started on 0x2a98a6b4a0, 
error_code: -1610612737
[E 12:58:43.318778] Flow proto 0x2a98a6b4a0 canceling a total of 2 BMI or Trove 
operations
[E 12:58:43.318851] Flow proto error cleanup finished 0x2a98a6b4a0, error_code: 
-1610612737
[E 11:50:38.093306]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 12:25:55.493550]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 11:52:11.051873] Flow proto error cleanup started on 0x2a971567e0, 
error_code: -1073741973
[E 11:52:11.052839] Flow proto 0x2a971567e0 canceling a total of 0 BMI or Trove 
operations
[E 11:52:11.052858] Flow proto error cleanup finished 0x2a971567e0, error_code: 
-1073741973
[E 00:37:00.609802] Job time out: cancelling flow operation, job_id: 60604133.
[E 00:37:00.610308] Flow proto cancel called on 0x2a9734b890
[E 00:37:21.082539] Flow proto error cleanup started on 0x2a9734b890, 
error_code: -1610612737
[E 00:37:21.084859] Flow proto 0x2a9734b890 canceling a total of 2 BMI or Trove 
operations
[E 00:37:21.084975] Flow proto error cleanup finished 0x2a9734b890, error_code: 
-1610612737
[E 09:02:28.364450]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 04:44:49.757475] Flow proto error cleanup started on 0x2a9abc37c0, 
error_code: -104
[E 04:44:49.758771] Flow proto 0x2a9abc37c0 canceling a total of 1 BMI or Trove 
operations
[E 04:44:49.758791] Flow proto error cleanup finished 0x2a9abc37c0, error_code: 
-104
[E 10:38:09.247852] Flow proto error cleanup started on 0x2a9db1e680, 
error_code: -1073741973
[E 10:38:09.248487] Flow proto 0x2a9db1e680 canceling a total of 1 BMI or Trove 
operations
[E 10:38:09.248525] Flow proto error cleanup finished 0x2a9db1e680, error_code: 
-1073741973
[E 20:07:03.036272] Flow proto error cleanup started on 0x2aa260b930, 
error_code: -1073741973
[E 20:07:03.037042] Flow proto 0x2aa260b930 canceling a total of 1 BMI or Trove 
operations
[E 20:07:03.037067] Flow proto error cleanup finished 0x2aa260b930, error_code: 
-1073741973

==> ff010/var/log/pvfs2-server.log <==
[E 12:57:45.662629] Flow proto cancel called on 0x2a98c771f0
[E 12:58:37.177711] Flow proto error cleanup started on 0x2a98c771f0, 
error_code: -1610612737
[E 12:58:37.178690] Flow proto 0x2a98c771f0 canceling a total of 2 BMI or Trove 
operations
[E 12:58:37.178771] Flow proto error cleanup finished 0x2a98c771f0, error_code: 
-1610612737
[E 11:41:06.579585] Job time out: cancelling flow operation, job_id: 261075524.
[E 11:41:06.580350] Flow proto cancel called on 0x12a1550
[E 11:41:17.661067] Flow proto error cleanup started on 0x12a1550, error_code: 
-1610612737
[E 11:41:17.663997] Flow proto 0x12a1550 canceling a total of 2 BMI or Trove 
operations
[E 11:41:17.664514] Flow proto error cleanup finished 0x12a1550, error_code: 
-1610612737
[E 11:47:05.893222] Job time out: cancelling flow operation, job_id: 261077518.
[E 11:47:05.893890] Flow proto cancel called on 0x1283ba0
[E 11:47:05.893926] Flow proto error cleanup started on 0x1283ba0, error_code: 
-1610612737
[E 11:48:23.943123] Flow proto 0x1283ba0 canceling a total of 1 BMI or Trove 
operations
[E 11:48:23.943696] Flow proto error cleanup finished 0x1283ba0, error_code: 
-1610612737
[E 11:50:40.247367]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 12:25:58.400013]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 11:52:11.007555] Flow proto error cleanup started on 0x2a96eefff0, 
error_code: -1073741973
[E 11:52:11.008692] Flow proto 0x2a96eefff0 canceling a total of 0 BMI or Trove 
operations
[E 11:52:11.008712] Flow proto error cleanup finished 0x2a96eefff0, error_code: 
-1073741973
[E 00:37:00.590231] Job time out: cancelling flow operation, job_id: 60722692.
[E 00:37:00.590849] Flow proto cancel called on 0x2a97326e90
[E 00:37:19.214512] Flow proto error cleanup started on 0x2a97326e90, 
error_code: -1610612737
[E 00:37:19.217037] Flow proto 0x2a97326e90 canceling a total of 2 BMI or Trove 
operations
[E 00:37:19.217121] Flow proto error cleanup finished 0x2a97326e90, error_code: 
-1610612737
[E 09:02:30.496085]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 11:14:02.909857] Flow proto error cleanup started on 0xd83060, error_code: 
-1073741973
[E 11:14:02.910777] Flow proto 0xd83060 canceling a total of 0 BMI or Trove 
operations
[E 11:14:02.910797] Flow proto error cleanup finished 0xd83060, error_code: 
-1073741973
[E 04:44:49.757205] Flow proto error cleanup started on 0x1902ef0, error_code: 
-104
[E 04:44:49.758324] Flow proto 0x1902ef0 canceling a total of 1 BMI or Trove 
operations
[E 04:44:49.758344] Flow proto error cleanup finished 0x1902ef0, error_code: 
-104
[E 10:38:12.786615] Flow proto error cleanup started on 0x2a9e17bd40, 
error_code: -1073741973
[E 10:38:12.787231] Flow proto 0x2a9e17bd40 canceling a total of 1 BMI or Trove 
operations
[E 10:38:12.787269] Flow proto error cleanup finished 0x2a9e17bd40, error_code: 
-1073741973
[E 20:06:58.044763] Flow proto error cleanup started on 0x2aa37d9600, 
error_code: -1073741973
[E 20:06:58.045488] Flow proto 0x2aa37d9600 canceling a total of 1 BMI or Trove 
operations
[E 20:06:58.045512] Flow proto error cleanup finished 0x2aa37d9600, error_code: 
-1073741973

==> ff011/var/log/pvfs2-server.log <==
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 08:34:03.160140] Flow proto error cleanup started on 0x2a971ea2b0, 
error_code: -104
[E 08:34:03.160818] Flow proto 0x2a971ea2b0 canceling a total of 1 BMI or Trove 
operations
[E 08:34:03.160838] Flow proto error cleanup finished 0x2a971ea2b0, error_code: 
-104
[E 08:36:52.014916] Job time out: cancelling flow operation, job_id: 43584119.
[E 08:36:52.015589] Flow proto cancel called on 0xd7e710
[E 08:36:56.733708] Flow proto error cleanup started on 0xd7e710, error_code: 
-1610612737
[E 08:36:56.734636] Flow proto 0xd7e710 canceling a total of 2 BMI or Trove 
operations
[E 08:36:56.734749] Flow proto error cleanup finished 0xd7e710, error_code: 
-1610612737
[E 11:29:25.143922] Job time out: cancelling flow operation, job_id: 93291033.
[E 11:29:25.144571] Flow proto cancel called on 0x2a99107a90
[E 11:30:03.333847] Flow proto error cleanup started on 0x2a99107a90, 
error_code: -1610612737
[E 11:30:03.335664] Flow proto 0x2a99107a90 canceling a total of 2 BMI or Trove 
operations
[E 11:30:03.335726] Flow proto error cleanup finished 0x2a99107a90, error_code: 
-1610612737
[E 12:57:45.882066] Job time out: cancelling flow operation, job_id: 174098170.
[E 12:57:45.882894] Flow proto cancel called on 0x2a988d06d0
[E 12:58:44.439322] Flow proto error cleanup started on 0x2a988d06d0, 
error_code: -1610612737
[E 12:58:44.440301] Flow proto 0x2a988d06d0 canceling a total of 2 BMI or Trove 
operations
[E 12:58:44.440381] Flow proto error cleanup finished 0x2a988d06d0, error_code: 
-1610612737
[E 11:50:42.315431]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 12:26:00.403797]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 11:52:11.008605] Flow proto error cleanup started on 0x2a98154cc0, 
error_code: -1073741973
[E 11:52:11.010067] Flow proto 0x2a98154cc0 canceling a total of 0 BMI or Trove 
operations
[E 11:52:11.010087] Flow proto error cleanup finished 0x2a98154cc0, error_code: 
-1073741973
[E 09:02:32.419305]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 11:14:02.910124] Flow proto error cleanup started on 0x2a98297510, 
error_code: -1073741973
[E 11:14:02.911096] Flow proto 0x2a98297510 canceling a total of 0 BMI or Trove 
operations
[E 11:14:02.911115] Flow proto error cleanup finished 0x2a98297510, error_code: 
-1073741973
[E 04:44:49.758449] Flow proto error cleanup started on 0x2a9af08d50, 
error_code: -104
[E 04:44:49.759700] Flow proto 0x2a9af08d50 canceling a total of 1 BMI or Trove 
operations
[E 04:44:49.759720] Flow proto error cleanup finished 0x2a9af08d50, error_code: 
-104
[E 10:38:07.559495] Flow proto error cleanup started on 0x2a9df4d8b0, 
error_code: -1073741973
[E 10:38:07.560125] Flow proto 0x2a9df4d8b0 canceling a total of 1 BMI or Trove 
operations
[E 10:38:07.560150] Flow proto error cleanup finished 0x2a9df4d8b0, error_code: 
-1073741973
[E 20:07:13.662001] Flow proto error cleanup started on 0x2aa2f658f0, 
error_code: -1073741973
[E 20:07:13.662675] Flow proto 0x2aa2f658f0 canceling a total of 1 BMI or Trove 
operations
[E 20:07:13.662703] Flow proto error cleanup finished 0x2aa2f658f0, error_code: 
-1073741973

==> ff012/var/log/pvfs2-server.log <==
[E 11:22:11.302587] dbpf_finalize: No such file or directory
[E 08:34:03.160263] Flow proto error cleanup started on 0x2a9738d7e0, 
error_code: -104
[E 08:34:03.161145] Flow proto 0x2a9738d7e0 canceling a total of 1 BMI or Trove 
operations
[E 08:34:03.161165] Flow proto error cleanup finished 0x2a9738d7e0, error_code: 
-104
[E 08:36:52.825555] Job time out: cancelling flow operation, job_id: 43493526.
[E 08:36:52.826176] Flow proto cancel called on 0x2a97376500
[E 08:36:56.949573] Flow proto error cleanup started on 0x2a97376500, 
error_code: -1610612737
[E 08:36:56.950684] Flow proto 0x2a97376500 canceling a total of 2 BMI or Trove 
operations
[E 08:36:56.950821] Flow proto error cleanup finished 0x2a97376500, error_code: 
-1610612737
[E 11:29:25.947687] Job time out: cancelling flow operation, job_id: 93095488.
[E 11:29:25.948328] Flow proto cancel called on 0x14209f0
[E 11:30:03.829131] Flow proto error cleanup started on 0x14209f0, error_code: 
-1610612737
[E 11:30:03.830901] Flow proto 0x14209f0 canceling a total of 2 BMI or Trove 
operations
[E 11:30:03.830976] Flow proto error cleanup finished 0x14209f0, error_code: 
-1610612737
[E 12:57:45.976144] Job time out: cancelling flow operation, job_id: 174239452.
[E 12:57:45.977003] Flow proto cancel called on 0x2a98b3d0b0
[E 12:58:37.054246] Flow proto error cleanup started on 0x2a98b3d0b0, 
error_code: -1610612737
[E 12:58:37.055212] Flow proto 0x2a98b3d0b0 canceling a total of 2 BMI or Trove 
operations
[E 12:58:37.055292] Flow proto error cleanup finished 0x2a98b3d0b0, error_code: 
-1610612737
[E 11:50:44.409229]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 12:26:01.613807]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 11:52:11.015067] Flow proto error cleanup started on 0x2a971e8bd0, 
error_code: -1073741973
[E 11:52:11.016821] Flow proto 0x2a971e8bd0 canceling a total of 0 BMI or Trove 
operations
[E 11:52:11.016847] Flow proto error cleanup finished 0x2a971e8bd0, error_code: 
-1073741973
[E 12:41:07.917434] Flow proto error cleanup started on 0x2a9771d180, 
error_code: -1073741973
[E 12:41:07.917915] Flow proto 0x2a9771d180 canceling a total of 0 BMI or Trove 
operations
[E 12:41:07.917934] Flow proto error cleanup finished 0x2a9771d180, error_code: 
-1073741973
[E 09:02:34.417028]
PVFS2 server got signal 15 (server_status_flag: 262143)
[E 04:44:49.758474] Flow proto error cleanup started on 0x2a9ad31b10, 
error_code: -104
[E 04:44:49.759557] Flow proto 0x2a9ad31b10 canceling a total of 1 BMI or Trove 
operations
[E 04:44:49.759578] Flow proto error cleanup finished 0x2a9ad31b10, error_code: 
-104
[E 10:38:11.060375] Flow proto error cleanup started on 0x2a9dd0a2c0, 
error_code: -1073741973
[E 10:38:11.061020] Flow proto 0x2a9dd0a2c0 canceling a total of 1 BMI or Trove 
operations
[E 10:38:11.061044] Flow proto error cleanup finished 0x2a9dd0a2c0, error_code: 
-1073741973
[E 20:07:04.830849] Flow proto error cleanup started on 0x2aa2e753f0, 
error_code: -1073741973
[E 20:07:04.831496] Flow proto 0x2aa2e753f0 canceling a total of 1 BMI or Trove 
operations
[E 20:07:04.831523] Flow proto error cleanup finished 0x2aa2e753f0, error_code: 
-1073741973



--
Brian D. Haymore
University of Utah
Center for High Performance Computing
155 South 1452 East RM 405
Salt Lake City, Ut 84112-0190
Phone: (801) 558-1150, Fax: (801) 585-5366
http://www.map.utah.edu/umaplink/0019.html

--- Begin Message ---

Molpro is a quantum package detailed at www.molpro.net 
I am working with a serial installation of 2006.1 version.

With either /scratch/serial or /tmp all 271 tests pass.
With /scratch/parallel all fail right at the start (259 failed, not sure
what is going on with remaining 12 as they list as successful but there
is no output) with:


Write error in iow_direct_write; fd=9, l=4, p=4096; write returns -1
This may indicate a filled disk, or that a disk quota has been exceeded

 ERROR WRITING            4 WORDS AT OFFSET       4096. TO FILE 1
IMPLEMENTATION=df   FILE HANDLE=     9  IERR=   -32
 ? Error
 ? I/O error
 ? The problem occurs in writew

 ERROR EXIT
 CURRENT STACK:      MAIN

 
Anita

********************************
Anita M. Orendt
Staff Scientist for Molecular Sciences
Center for High Performance Computing
University of Utah
155 South 1452 East Room 422
Salt Lake City, UT 84112-0190
 
email:  [EMAIL PROTECTED]
phone: 801-231-2762
fax: 801-585-5366
*********************************
 



--- End Message ---


_______________________________________________
Pvfs2-developers mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers






















					Reply via email to