Venu,
Sorry I had a copy-and-paste error on the last
email. I didn't show the correct ip_input()
probe output. It's now below.
Dan
ip_input()
==========
CPU FUNCTION
0 -> ip_input lport 56618 dport 23
seq 1429020322 ack 0 flags 2
0 -> ip_input lport 56618 dport 23
seq 1429020322 ack 0 flags 2
0 -> ip_input lport 56618 dport 23
seq 1429020322 ack 0 flags 2
0 -> ip_input lport 56618 dport 23
seq 1429020322 ack 0 flags 2
0 -> ip_input lport 56618 dport 23
seq 1429020322 ack 0 flags 2
0 -> ip_input lport 23 dport 56618
seq 3146400406 ack 1429020323 flags 12
0 -> ip_input lport 56618 dport 23
seq 1429020347 ack 3487148167 flags 10
0 -> ip_input lport 56618 dport 23
seq 1429020323 ack 3487148167 flags 18
0 -> ip_input lport 56618 dport 23
seq 1429020347 ack 3487148167 flags 10
0 -> ip_input lport 56618 dport 23
seq 1429020323 ack 3487148167 flags 18
0 -> ip_input lport 56618 dport 23
seq 1429020347 ack 3487148167 flags 10
0 -> ip_input lport 56618 dport 23
seq 1429020323 ack 3487148167 flags 18
0 -> ip_input lport 56618 dport 23
seq 1429020347 ack 3487148167 flags 10
0 -> ip_input lport 56618 dport 23
seq 1429020323 ack 3487148167 flags 18
ip_tcp_input()
==============
CPU FUNCTION
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020322 ack 0 flags 2
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020322 ack 0 flags 2
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020322 ack 0 flags 2
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020322 ack 0 flags 2
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020322 ack 0 flags 2
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020347 ack 3487148167 flags 10
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020323 ack 3487148167 flags 18
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020347 ack 3487148167 flags 10
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020323 ack 3487148167 flags 18
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020347 ack 3487148167 flags 10
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020323 ack 3487148167 flags 18
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020347 ack 3487148167 flags 10
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020323 ack 3487148167 flags 18
-----Original Message-----
>From: Dan Celidonio <[EMAIL PROTECTED]>
>Sent: Aug 3, 2006 7:15 PM
>To: venugopal iyer <[EMAIL PROTECTED]>
>Cc: [email protected]
>Subject: Re: [networking-discuss] problem with GLDV3 bge interface
>
>Venu,
>
>I ran dtrace probes at ip_input() and ip_tcp_input()
>I do see the ACK for port 23 (the telnet port).
>
>>From the previous probe at tcp_rput_data() this
>ACK is not seen.
>
>Dan
>
>ip_input()
>==========
>CPU FUNCTION
> 0 -> ip_tcp_input lport 56618 dport 23
> seq 1429020322 ack 0 flags 2
>
> 0 -> ip_tcp_input lport 56618 dport 23
> seq 1429020322 ack 0 flags 2
>
> 0 -> ip_tcp_input lport 56618 dport 23
> seq 1429020322 ack 0 flags 2
>
> 0 -> ip_tcp_input lport 56618 dport 23
> seq 1429020322 ack 0 flags 2
>
> 0 -> ip_tcp_input lport 56618 dport 23
> seq 1429020322 ack 0 flags 2
>
> 0 -> ip_tcp_input lport 56618 dport 23
> seq 1429020347 ack 3487148167 flags 10
>
> 0 -> ip_tcp_input lport 56618 dport 23
> seq 1429020323 ack 3487148167 flags 18
>
> 0 -> ip_tcp_input lport 56618 dport 23
> seq 1429020347 ack 3487148167 flags 10
>
> 0 -> ip_tcp_input lport 56618 dport 23
> seq 1429020323 ack 3487148167 flags 18
>
> 0 -> ip_tcp_input lport 56618 dport 23
> seq 1429020347 ack 3487148167 flags 10
>
> 0 -> ip_tcp_input lport 56618 dport 23
> seq 1429020323 ack 3487148167 flags 18
>
> 0 -> ip_tcp_input lport 56618 dport 23
> seq 1429020347 ack 3487148167 flags 10
>
> 0 -> ip_tcp_input lport 56618 dport 23
> seq 1429020323 ack 3487148167 flags 18
>
>
>ip_tcp_input()
>==============
>CPU FUNCTION
> 0 -> ip_tcp_input lport 56618 dport 23
> seq 1429020322 ack 0 flags 2
>
> 0 -> ip_tcp_input lport 56618 dport 23
> seq 1429020322 ack 0 flags 2
>
> 0 -> ip_tcp_input lport 56618 dport 23
> seq 1429020322 ack 0 flags 2
>
> 0 -> ip_tcp_input lport 56618 dport 23
> seq 1429020322 ack 0 flags 2
>
> 0 -> ip_tcp_input lport 56618 dport 23
> seq 1429020322 ack 0 flags 2
>
> 0 -> ip_tcp_input lport 56618 dport 23
> seq 1429020347 ack 3487148167 flags 10
>
> 0 -> ip_tcp_input lport 56618 dport 23
> seq 1429020323 ack 3487148167 flags 18
>
> 0 -> ip_tcp_input lport 56618 dport 23
> seq 1429020347 ack 3487148167 flags 10
>
> 0 -> ip_tcp_input lport 56618 dport 23
> seq 1429020323 ack 3487148167 flags 18
>
> 0 -> ip_tcp_input lport 56618 dport 23
> seq 1429020347 ack 3487148167 flags 10
>
> 0 -> ip_tcp_input lport 56618 dport 23
> seq 1429020323 ack 3487148167 flags 18
>
> 0 -> ip_tcp_input lport 56618 dport 23
> seq 1429020347 ack 3487148167 flags 10
>
> 0 -> ip_tcp_input lport 56618 dport 23
> seq 1429020323 ack 3487148167 flags 18
>
>
>
>-----Original Message-----
>>From: venugopal iyer <[EMAIL PROTECTED]>
>>Sent: Aug 3, 2006 6:28 PM
>>To: Dan Celidonio <[EMAIL PROTECTED]>
>>Cc: [email protected]
>>Subject: Re: [networking-discuss] problem with GLDV3 bge interface
>>
>>
>>Dan:
>>
>>On Thu, 3 Aug 2006, Dan Celidonio wrote:
>>
>>> Hi Venu,
>>>
>>> Thanks for your response.
>>>
>>> I tried your dtrace scripts for both tcp_conn_request()
>>> and tcp_rput_data() twice on the server node. The first
>>> time I del'ed out of the tcp_rput_data dtrace before the
>>> client terminated the connection. The secind time I let
>>> it go until the connection was terminated by the client.
>>>
>>> This test is just a simple telnet.
>>>
>>> I see this on the client:
>>>
>>> Connected to xxx.xxx.xxx.xx. [actual IP not shown]
>>> Escape character is '^]'.
>>> Connection closed by foreign host.
>>>
>>> Looking at the below results it seems that tcp_rput_data()
>>> never sees the ACK sent by the client. However, I do see the
>>> ACK come in from the client in our Streams module before
>>> we send it up the IP stack. We send it up to IP via a
>>> call to ip_input(). I'll need to do more digging.
>>
>>I'd check ip_tcp_input().
>>
>>>
>>> Thanks,
>>> Dan
>>>
>>> 1st try
>>> =======
>>> [There was just one print in tcp_conn_request()]
>>
>>That's right, only the SYN comes in here, rest should be in
>>tcp_rput_data().
>>
>>>
>>> CPU FUNCTION
>>> 0 -> tcp_conn_request TCP 30002e13600 state -3 suna 0 snxt 0 rnxt 0
>>> lport 52203 dport 23
>>> seq 2476877873 ack 0 flags 2
>>>
>>> [There were many prints in tcp_rput_data()]
>>> I removed all prints which didn't have port 23 printed.
>>
>>
>>You can make the dtrace probe conditional on destination port 23, i.e.
>>using my prev example, something like:
>>
>>fbt::tcp_conn_request:entry
>>/ *(uint16_t *)(((tcph_t *)((uchar_t *)(ipha_t *)((mblk_t *)arg1)->b_rptr +
>>20))->th_fport) == 23 /
>>{
>> ....
>>}
>>
>>
>>> I terminated the dtrace before the client terminated
>>> the connection.
>>>
>>> CPU FUNCTION
>>> 0 -> tcp_rput_data TCP 30002dfe9c0 state -1 suna
>>> 614917337 snxt 614917338 rnxt 2476877874
>>> lport 52203 dport 23
>>> seq 2476877873 ack 0 flags 2
>>>
>>> 0 -> tcp_rput_data TCP 30002dfe9c0 state -1 suna
>>> 614917337 snxt 614917338 rnxt 2476877874
>>> lport 52203 dport 23
>>> seq 2476877873 ack 0 flags 2
>>>
>>> 0 -> tcp_rput_data TCP 30002dfe9c0 state -1 suna
>>> 614917337 snxt 614917338 rnxt 2476877874
>>> lport 52203 dport 23
>>> seq 2476877873 ack 0 flags 2
>>>
>>> 0 -> tcp_rput_data TCP 30002dfe9c0 state -1 suna
>>> 614917337 snxt 614917338 rnxt 2476877874
>>> lport 52203 dport 23
>>> seq 2476877873 ack 0 flags 2
>>
>>
>>From the above looks like we keep getting SYN (flags = 2). Do you see these
>>SYN in the snoop as well? Does the other side get the SYN-ACK (and send the
>>ACK)?
>>
>>>
>>> 2nd try
>>> =======
>>> [There was just one print in tcp_conn_request()]
>>>
>>> CPU FUNCTION
>>> 0 -> tcp_conn_request TCP 30002e13600 state -3 suna 0 snxt 0 rnxt
>>> lport 52463 dport 23
>>> seq 2667176845 ack 0 flags 2
>>>
>>> [There were many prints in tcp_rput_data()]
>>> I removed all prints which didn't have port 23 printed.
>>> I let it run until the telnet connection was closed by
>>> the client.
>>>
>>> CPU FUNCTION
>>> 0 -> tcp_rput_data TCP 30002dfd100 state -1 suna
>>> 774266961 snxt 774266962 rnxt 2667176846
>>> lport 52463 dport 23
>>> seq 2667176845 ack 0 flags 2
>>>
>>> 0 -> tcp_rput_data TCP 30002dfd100 state -1 suna
>>> 774266961 snxt 774266962 rnxt 2667176846
>>> lport 52463 dport 23
>>> seq 2667176845 ack 0 flags 2
>>>
>>> 0 -> tcp_rput_data TCP 30002dfd100 state -1 suna
>>> 774266961 snxt 774266962 rnxt 2667176846
>>> lport 52463 dport 23
>>> seq 2667176845 ack 0 flags 2
>>>
>>> 0 -> tcp_rput_data TCP 30002dfd100 state -1 suna
>>> 774266961 snxt 774266962 rnxt 2667176846
>>> lport 52463 dport 23
>>> seq 2667176845 ack 0 flags 2
>>>
>>> 0 -> tcp_rput_data TCP 30002dfd100 state -1 suna
>>> 774266961 snxt 774266962 rnxt 2667176846
>>> lport 52463 dport 23
>>> seq 2667176870 ack 4250440928 flags 10
>>
>>this probably causes bad_ack... since the ack is greater than tcp_snxt.
>>(Look at the comments in tcp_rput_data, LL12779-12790).
>>
>>-venu
>>
>>>
>>> 0 -> tcp_rput_data TCP 30002dfd100 state -1 suna
>>> 774266961 snxt 774266962 rnxt 2667176846
>>> lport 52463 dport 23
>>> seq 2667176846 ack 4250440928 flags 18
>>>
>>> 0 -> tcp_rput_data TCP 30002dfd100 state -1 suna
>>> 774266961 snxt 774266962 rnxt 2667176846
>>> lport 52463 dport 23
>>> seq 2667176870 ack 4250440928 flags 10
>>>
>>> 0 -> tcp_rput_data TCP 30002dfd100 state -1 suna
>>> 774266961 snxt 774266962 rnxt 2667176846
>>> lport 52463 dport 23
>>> seq 2667176846 ack 4250440928 flags 18
>>>
>>> 0 -> tcp_rput_data TCP 30002dfd100 state -1 suna
>>> 774266961 snxt 774266962 rnxt 2667176846
>>> lport 52463 dport 23
>>> seq 2667176870 ack 4250440928 flags 10
>>>
>>> 0 -> tcp_rput_data TCP 30002dfd100 state -1 suna
>>> 774266961 snxt 774266962 rnxt 2667176846
>>> lport 52463 dport 23
>>> seq 2667176846 ack 4250440928 flags 18
>>>
>>> 0 -> tcp_rput_data TCP 30002dfd100 state -1 suna
>>> 774266961 snxt 774266962 rnxt 2667176846
>>> lport 52463 dport 23
>>> seq 2667176870 ack 4250440928 flags 10
>>>
>>> 0 -> tcp_rput_data TCP 30002dfd100 state -1 suna
>>> 774266961 snxt 774266962 rnxt 2667176846
>>> lport 52463 dport 23
>>> seq 2667176846 ack 4250440928 flags 18
>>>
>>>
>>> -----Original Message-----
>>>> From: venugopal iyer <[EMAIL PROTECTED]>
>>>> Sent: Aug 3, 2006 2:35 PM
>>>> To: Dan Celidonio <[EMAIL PROTECTED]>
>>>> Cc: [email protected]
>>>> Subject: Re: [networking-discuss] problem with GLDV3 bge interface
>>>>
>>>>
>>>>
>>>> [...]
>>>>
>>>>> Some of the problems seen on the server using the
>>>>> newly converted load balancer driver for bge GLDV3
>>>>> interface are below. We set tcp_trace and tcp_debug and use
>>>>> strace to see these errors on the server node.
>>>>>
>>>>> 1. The SYN packet coming from the client into the
>>>>> traffic cop node and forwarded to the server node
>>>>> is getting an "unacceptable sequence number gap"
>>>>> error from the TCP layer on the server.
>>>>>
>>>>> 2. TCP Duplicate segment counter increments
>>>>> from the TCP layer on the server.
>>>>>
>>>>> 3. TCP Duplicate Ack counter increments
>>>>> from the TCP layer on the server.
>>>>>
>>>>> 4. "bad_ack" in SYN_RCVD state.
>>>>
>>>> To get a sense for what's happening in TCP, I'd use dtrace in
>>>> tcp_conn_request() and tcp_rput_data() and print out the tcp_t
>>>> (get this from the conn_t in arg0) along with its state, snxt,
>>>> suna and rnxt. Additionally, also print out the TCP header from
>>>> the incoming packet (arg1), the seq, ack and flags, something
>>>> like:
>>>>
>>>> fbt::tcp_conn_request:entry
>>>> {
>>>> connp = (conn_t *)arg0;
>>>> tcp = (tcp_t *)connp->conn_tcp;
>>>> ipha = (ipha_t *)((mblk_t *)arg1)->b_rptr;
>>>> /* 20 - assuming no IP options etc. */
>>>> tcph = (tcph_t *)((uchar_t *)ipha + 20);
>>>>
>>>> printf("\tTCP %p state %d suna %u snxt %u rnxt %u\n",
>>>> tcp, tcp->tcp_state, tcp->tcp_suna, tcp->tcp_snxt,
>>>> tcp->tcp_rnxt);
>>>> printf("\tlport %u dport %u\n", *(uint16_t *)tcph->th_lport,
>>>> *(uint16_t *)tcph->th_fport);
>>>> printf("\tseq %u ack %u flags %x\n",
>>>> *(uint32_t *)tcph->th_seq, *(uint32_t *)tcph->th_ack,
>>>> *tcph->th_flags);
>>>> }
>>>>
>>>> similarly for tcp_rput_data.
>>>>
>>>> (on an x86 machine you would need to print out the TCP header content
>>>> taking endianness into consideration).
>>>>
>>>> I suppose you could restrict the dtrace probes to the interested port.
>>>>
>>>> Ordinarily, I'd expect the SYN to come into tcp_conn_request(),
>>>> which will result in sending the SYN-ACK and transitioning to
>>>> SYN-RCVD. The ACK from the client will be processed in tcp_rput_data.
>>>>
>>>> You can check the code in tcp_rput_data() that result in the
>>>> messages/conditions that you list above.
>>>>
>>>> Look at:
>>>> http://cvs.opensolaris.org/source/xref/on/usr/src/uts/common/inet/tcp.h
>>>>
>>>> for the TCP structure (tcp_t) and TCP header structure (tcph_t)
>>>>
>>>> Look at:
>>>>
>>>> http://cvs.opensolaris.org/source/xref/on/usr/src/uts/common/inet/tcp/tcp.c
>>>>
>>>> for tcp_conn_request() and tcp_rput_data() - L12764-12789, L12903-12961
>>>> etc.
>>>>
>>>> -venu
>>>>
>>>>
>>>>>
>>>>> Thanks,
>>>>> Dan
>>>>>
>>>>>
>>>>> This message posted from opensolaris.org
>>>>> _______________________________________________
>>>>> networking-discuss mailing list
>>>>> [email protected]
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>_______________________________________________
>networking-discuss mailing list
>[email protected]
_______________________________________________
networking-discuss mailing list
[email protected]
