I don't have any generic, simple solutions to do this. As you noted,
it can be much more difficult in Solaris to trace things that are
event driven or timer driven then it is to trace things starting at
the syscall level.
The next step I did on my problem was to capture the queue pointer
used in the call, and then started tracing all function calls that
take queue pointers that had the interesting queue pointer.
Unfortunately, that can only get you so far, because the queue pointer
for the other end is different. But you can find that out using mdb
and do the same thing at that end.
Good luck.
Iwan Aucamp wrote:
Hi Brian
My Dilemma is this, I cant trace inside accept(2) the same way i did
here because Solaris completes TCP handshaking regardless of accept(2)
(given there is sufficent queue space, which there is supposed to be here).
I tried tracing all FBT (no predicates) on a test system while doing a
connect so I have a reference to look at but when doing this DTRACE
drops allot and the output is largely useless - and doing this on the
live system is out of question really.
Any ideas how to minimize the coverage of dtrace script to run on listner ?
Anyway, thanks for the hint though, will check the code - this sounds
quite plausible.
Regards
-------- Original Message --------
Subject: Re: [networking-discuss] connect(2) blocks for more than 250 ms
over loopback
From: Brian Utterback <[email protected]>
To: Iwan Aucamp <[email protected]>
Cc: [email protected]
Date: Tue Sep 22 2009 19:23:35 GMT+0200 (SAST)
I am afraid that this dtrace output isn't going to be useful. The
problem is happening at the other end of the connection, in the accept
processing. When you do a connect, you send a message down the stream
saying that you want to connect. Often, the thread responsible for
processing the connect will continue the processing down the stream
and do the accept side processing as well, but sometimes the stream
will be locked because another thread is processing in it. In that
case, the connect message is queued and the connect thread goes to
sleep, waiting for a response. Unfortunately, in your case the
response doesn't seem to be forthcoming, and the signal wakes it up
and aborts the connect call.
I saw something similar to what you are experiencing once. It turned
out that the program incorrectly re-used sockets after failed
connection attempts. Since the sockets had some old state when connect
was called, they failed to transition through the states correctly and
ended up waiting for the connect to complete, even though the connect
message was never sent. You might want to check that.
Iwan Aucamp wrote:
Hi
On a T2000 node running Solaris 10 u5 connect(2) over loopback to an
application which is completely idle blocks for longer than 250 ms.
The system load averages when this occurs is also below 2 and another
system (same os, hardware and software) with much higher CPU load
does have this problem.
In an attempt to figure out what is causing this I ran dtrace with
fbt:: and syscall:connect:, extracting all fbt:: when a connect fails
(application calling connect times out after 250ms using alarm so
connect fails with EINTR).
The results of this is attached:
connect.ok.verbose : successfull connect, verbose, for reference
connect.ok : successfull connect, for reference (can
be used with diff against connect.timeout)
connect.timeout.verbose : timed-out connect, verbose
connect.timeout : timed-out connect (can be used with diff
against connect.ok)
The major difference starts at:
connect.timeout:
246: -> (fbt:ip:1:ip_wput_local:entry)
247: -> (fbt:ip:1:ip_fanout_tcp:entry)
250: ->
(fbt:ip:1:squeue_enter_nodrain:entry)
connect.ok:
240: -> (fbt:ip:1:ip_wput_local:entry)
241: -> (fbt:ip:1:ip_fanout_tcp:entry)
244: ->
(fbt:ip:1:squeue_enter_nodrain:entry)
From here
connect.timeout goes to fbt:genunix:timeout and
fbt:genunix:timeout_common.
As far as i can figure a TCP SYN packet is sent over IP but a TCP
SYN-ACK never comes back.
Does this seem like the correct interpretation, and does anybody have
any ideas regarding this ?
Regards
------------------------------------------------------------------------
_______________________________________________
networking-discuss mailing list
[email protected]
--
blu
It's bad civic hygiene to build technologies that could someday be
used to facilitate a police state. - Bruce Schneier
----------------------------------------------------------------------
Brian Utterback - Solaris RPE, Sun Microsystems, Inc.
Ph:877-259-7345, Em:brian.utterback-at-ess-you-enn-dot-kom
_______________________________________________
networking-discuss mailing list
[email protected]
