I do not know if this is related or not, but when I try to ^C the
server, I do not see the normal shutdown messages such as:
PVFS2 server got signal 2 (server_status_flag: 262143)
[D 01/11 13:40] *** server shutdown in progress ***
[D 01/11 13:40] [+] halting state machine processor [ ... ]
[D 01/11 13:40] [-] state machine processor [ stopped ]
<snip>
Instead, I only get:
PVFS2 server got signal 2 (server_status_flag: 262143)
and nothing else. I have waited more than 5 minutes before ^Z the
process and killing it. Also, before using ^C, it is unresponsive to
new operations such as pvfs2-ls.
Scott
On Jan 11, 2007, at 1:48 PM, Scott Atchley wrote:
Well, I still can't use the core file but it is not happening in my
BMI_mx_cancel() function. I added print statements at several
locations including the end of the function and all print.
[E 13:36:40.025329] job_time_mgr_expire: job time out: cancelling
bmi operation, job_id: 71.
[D 13:36:40.025508] PINT_thread_mgr_bmi_cancel: trying to cancel
opid: 72, ptr: 0x810b924.
[D 13:36:40.025577] BMI_cancel: cancel id 72
* BMI_mx_cancel RX op_id 72 mxc_state 4 peer state 2
BMI_mx_cancel calling mx_cancel()
BMI_mx_cancel mx_cancel() succeeded
BMI_mx_cancel bmx_deq_pending_ctx()
BMI_mx_cancel bmx_q_canceled_ctx()
BMI_mx_cancel done
[3] + segmentation fault (core dumped) pvfs2-cp /scratch/atchley/
test /mnt/pvfs2/test-${I}
[E 13:36:40.988876] job_time_mgr_expire: job time out: cancelling
bmi operation, job_id: 77.
[D 13:36:40.988926] PINT_thread_mgr_bmi_cancel: trying to cancel
opid: 78, ptr: 0x810b924.
[D 13:36:40.988945] BMI_cancel: cancel id 78
* BMI_mx_cancel RX op_id 78 mxc_state 4 peer state 2
BMI_mx_cancel calling mx_cancel()
BMI_mx_cancel mx_cancel() succeeded
BMI_mx_cancel bmx_deq_pending_ctx()
BMI_mx_cancel bmx_q_canceled_ctx()
BMI_mx_cancel done
[2] + segmentation fault (core dumped) pvfs2-cp /scratch/atchley/
test /mnt/pvfs2/test-${I}
I noticed that bmi_ib does not actually cancel an operation, it
simply closes the connection. MX can cancel receives, and I only
close the connection if I need to cancel a send. I do not know if
this is relevant or not.
Scott
On Jan 11, 2007, at 11:58 AM, Scott Atchley wrote:
Hi Sam,
I am using a 256 MB file on a machine with only 1 GB of memory.
The server sees the timeouts after ~60 seconds and the client is
much longer (and may be 5 minutes). I will time it on my next run.
Scott
On Jan 11, 2007, at 11:48 AM, Sam Lang wrote:
Hi Scott,
How big is the test file you're copying? tcp doesn't hang with
two pvfs2-cp on a 40MB, but I should probably try something
larger. :-) The flow timeouts on the server are set to 5
minutes. Are you waiting that long before seeing those messages
in the log?
-sam
On Jan 11, 2007, at 10:41 AM, Scott Atchley wrote:
Hi all,
Here is a little more detail. On the server, after the stall I
only see:
[E 01/11 11:28] job_time_mgr_expire: job time out: cancelling
flow operation, job_id: 362.
[E 01/11 11:28] fp_multiqueue_cancel: flow proto cancel called
on 0x8197828
[D 01/11 11:28] fp_multiqueue_cancel: called on already
completed flow; doing nothing.
There are no timeouts in BMI.
On the client, I eventually see:
[E 11:32:21.004481] job_time_mgr_expire: job time out:
cancelling bmi operation, job_id: 41.
[D 11:32:21.004649] PINT_thread_mgr_bmi_cancel: trying to cancel
opid: 42, ptr: 0x810c5ac.
[D 11:32:21.004719] BMI_cancel: cancel id 42
* BMI_mx_cancel RX op_id 42 mxc_state 4 peer state 2
/* This is a recv that is pending (mxc_state 4) and the peer is
READY (peer state 2) */
[2] - segmentation fault (core dumped) pvfs2-cp /scratch/
atchley/test /mnt/pvfs2/test-${I}
[E 11:32:21.993383] job_time_mgr_expire: job time out:
cancelling bmi operation, job_id: 53.
[D 11:32:21.993421] PINT_thread_mgr_bmi_cancel: trying to cancel
opid: 54, ptr: 0x810c5ac.
[D 11:32:21.993439] BMI_cancel: cancel id 54
* BMI_mx_cancel RX op_id 54 mxc_state 4 peer state 2
/* This is a recv that is pending (mxc_state 4) and the peer is
READY (peer state 2) */
[3] + segmentation fault (core dumped) pvfs2-cp /scratch/
atchley/test /mnt/pvfs2/test-${I}
Since the receives are pending (posted, not completed), the
cancel should succeed. It is probably a bug in my code that
causes the segfaults. Unfortunately, the core files are not usable:
% gdb pvfs2-cp core.28788
"core.28788" is not a core dump: File format not recognized
File disagrees:
% file core.28788
core.28788: ELF 32-bit LSB core file Intel 80386, version 1
(SYSV), SVR4-style, SVR4-style, from 'pvfs2-cp'
If the other BMI methods do not hang, then I need to keep digging.
Scott
On Jan 11, 2007, at 11:23 AM, Scott Atchley wrote:
Hi all,
I am simply running two pvfs2-cp processes at the same time to
see how everything works. For some reason, the copies start but
do not finish. Eventually, I see timeouts in BMI but not in
bmi_mx. Before I spend too much time on this, can other methods
run two copies at the same time?
$ for I in 1 2 ; do
pvfs2-cp test /mnt/pvfs2/test-${I} &
done
Thanks,
Scott
_______________________________________________
Pvfs2-developers mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-
developers
_______________________________________________
Pvfs2-developers mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-
developers
_______________________________________________
Pvfs2-developers mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
_______________________________________________
Pvfs2-developers mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
_______________________________________________
Pvfs2-developers mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
