We changed our sockets to be blocking(both on the server and the client) and are no longer setting destination droppable to false. When running one client (on one node) and a server (on a different node) we saw a throughput of about 56,000 msgs/sec with our hardware. If we had two clients(on one node) and two servers(servers with the same TIPC Name both on a different node), the total throughput to the node that the servers is on is still around 56,000 msgs/sec and the throughput is split evenly between the servers.
In all cases I sent a total of 25 Million messages with a throttle of 100,000 msgs/sec. The output you see is printed every 10 seconds. Couple of more observations: Next Test: Setup: Client1(node 1) -> Server1 (Node 2) Client2(node 3) -> Server2 (Node 2) Server 1 and Server 2 have different TIPC Names. Any ideas on why we see such a difference in throughput on the two different links? The sum is still around 56,000 msgs/sec. Server 1: Msgs per second = 30826 Messages in the time diff = 308260 Total Msgs= 308260 Msgs per second = 46178 Messages in the time diff = 461780 Total Msgs= 770040 Msgs per second = 40708.5 Messages in the time diff = 407085 Total Msgs= 1177125 Msgs per second = 44332.2 Messages in the time diff = 443322 Total Msgs= 1620447 Msgs per second = 40013.1 Messages in the time diff = 400131 Total Msgs= 2020578 Msgs per second = 44903.7 Messages in the time diff = 449037 Total Msgs= 2469615 Msgs per second = 53762.3 Messages in the time diff = 537623 Total Msgs= 3007238 Msgs per second = 42374.3 Messages in the time diff = 423743 Total Msgs= 3430981 Msgs per second = 52915 Messages in the time diff = 529150 Total Msgs= 3960131 Msgs per second = 50055.5 Messages in the time diff = 500555 Total Msgs= 4460686 Msgs per second = 35875.1 Messages in the time diff = 358751 Total Msgs= 4819437 Msgs per second = 47139.7 Messages in the time diff = 471397 Total Msgs= 5290834 Msgs per second = 51357 Messages in the time diff = 513570 Total Msgs= 5804404 Server 2: Msgs per second = 2721.4 Messages in the time diff = 27214 Total Msgs= 27214 Msgs per second = 10614.5 Messages in the time diff = 106145 Total Msgs= 133359 Msgs per second = 15566.2 Messages in the time diff = 155662 Total Msgs= 289021 Msgs per second = 11808 Messages in the time diff = 118080 Total Msgs= 407101 Msgs per second = 16563.3 Messages in the time diff = 165633 Total Msgs= 572734 Msgs per second = 8827.1 Messages in the time diff = 88271 Total Msgs= 661005 Msgs per second = 5155.5 Messages in the time diff = 51555 Total Msgs= 712560 Msgs per second = 11713.7 Messages in the time diff = 117137 Total Msgs= 829697 Msgs per second = 5957 Messages in the time diff = 59570 Total Msgs= 889267 Msgs per second = 6401.6 Messages in the time diff = 64016 Total Msgs= 953283 Msgs per second = 17586.5 Messages in the time diff = 175865 Total Msgs= 1129148 Msgs per second = 9707.6 Messages in the time diff = 97076 Total Msgs= 1226224 Msgs per second = 7532.1 Messages in the time diff = 75321 Total Msgs= 1301545 ------------------------------------------------------------------------ ---------------- Next Test: I'm not sure I understand the results below. Server 1 receiving messages is not affected by the fact that client 2 is on the same node, but the throughput to server 2 is cut in half from the 56,000 msgs/sec apparently because of the fact that client2 is sending at the same time. Any thoughts? Client1(Node 3) -> Server1(Node 2) Client2(Node 2) -> Server2(Node 1) results: Node2 (Server 1 receiving): Msgs per second = 51563.5 Messages in the time diff = 515635 Total Msgs= 515635 Msgs per second = 55464.2 Messages in the time diff = 554642 Total Msgs= 1070277 Msgs per second = 55152.8 Messages in the time diff = 551528 Total Msgs= 1621805 Msgs per second = 55367.1 Messages in the time diff = 553671 Total Msgs= 2175476 Msgs per second = 55160.2 Messages in the time diff = 551602 Total Msgs= 2727078 Msgs per second = 55514.2 Messages in the time diff = 555142 Total Msgs= 3282220 Msgs per second = 55330.6 Messages in the time diff = 553306 Total Msgs= 3835526 Msgs per second = 55629.5 Messages in the time diff = 556295 Total Msgs= 4391821 Msgs per second = 55452.2 Messages in the time diff = 554522 Total Msgs= 4946343 Msgs per second = 55362.2 Messages in the time diff = 553622 Total Msgs= 5499965 Msgs per second = 55208.9 Messages in the time diff = 552089 Total Msgs= 6052054 Msgs per second = 55422.5 Messages in the time diff = 554225 Total Msgs= 6606279 Msgs per second = 55507.2 Messages in the time diff = 555072 Total Msgs= 7161351 Msgs per second = 55380 Messages in the time diff = 553800 Total Msgs= 7715151 Msgs per second = 55617.1 Messages in the time diff = 556171 Total Msgs= 8271322 Msgs per second = 55447 Messages in the time diff = 554470 Total Msgs= 8825792 Msgs per second = 55585.6 Messages in the time diff = 555856 Total Msgs= 9381648 Msgs per second = 55378.9 Messages in the time diff = 553789 Total Msgs= 9935437 Node 1 (Server 2 receiving): Msgs per second = 27768.8 Messages in the time diff = 277688 Total Msgs= 334396 Msgs per second = 25784.2 Messages in the time diff = 257842 Total Msgs= 592238 Msgs per second = 25504.7 Messages in the time diff = 255047 Total Msgs= 847285 Msgs per second = 25539 Messages in the time diff = 255390 Total Msgs= 1102675 Msgs per second = 25264.1 Messages in the time diff = 252641 Total Msgs= 1355316 Msgs per second = 25267.9 Messages in the time diff = 252679 Total Msgs= 1607995 Msgs per second = 25420.1 Messages in the time diff = 254201 Total Msgs= 1862196 Msgs per second = 24963.8 Messages in the time diff = 249638 Total Msgs= 2111834 Msgs per second = 25287.4 Messages in the time diff = 252874 Total Msgs= 2364708 Msgs per second = 25242.8 Messages in the time diff = 252428 Total Msgs= 2617136 Msgs per second = 25691.3 Messages in the time diff = 256913 Total Msgs= 2874049 Msgs per second = 25387.8 Messages in the time diff = 253878 Total Msgs= 3127927 Msgs per second = 25679.9 Messages in the time diff = 256799 Total Msgs= 3384726 Msgs per second = 25464.7 Messages in the time diff = 254647 Total Msgs= 3639373 Msgs per second = 25251.5 Messages in the time diff = 252515 Total Msgs= 3891888 Msgs per second = 25608.7 Messages in the time diff = 256087 Total Msgs= 4147975 Msgs per second = 25280 Messages in the time diff = 252800 Total Msgs= 4400775 Msgs per second = 25394.5 Messages in the time diff = 253945 Total Msgs= 4654720 Msgs per second = 25430.7 Messages in the time diff = 254307 Total Msgs= 4909027 Msgs per second = 25349.2 Messages in the time diff = 253492 Total Msgs= 5162519 Msgs per second = 25126.1 Messages in the time diff = 251261 Total Msgs= 5413780 Msgs per second = 25475.7 Messages in the time diff = 254757 Total Msgs= 5668537 -----Original Message----- From: Stephens, Allan [mailto:[EMAIL PROTECTED] Sent: Thursday, September 20, 2007 9:14 AM To: Randy Macleod; Nayman Felix-QA5535; [email protected] Subject: RE: [tipc-discussion] Link Congestion problem Hi Felix: I think Randy's analysis has pretty much said it all, so I don't really have anything to add. I'm a bit puzzled about why you're trying to use non-blocking sockets with non-discardable message traffic. The non-discardable message type suggests that you don't want to lose any messages, in which case you have no alternative but to wait when you are generating messages faster than they are being consumed. If there is some critical processing that needs to occur on a regular basis, and which would be hampered by a blocking send operation, maybe this processing should be handled by a different thread of control ... Regards, Al > -----Original Message----- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of > Randy Macleod > Sent: Wednesday, September 19, 2007 9:14 PM > To: Nayman Felix-QA5535; [email protected] > Subject: Re: [tipc-discussion] Link Congestion problem > > Hi! > > On 9/19/07, Nayman Felix-QA5535 <[EMAIL PROTECTED]> wrote: > > > > I'm running TIPC 1.5.12 on a linux 2.6.9 kernel on two > different nodes > > and I'm seeing a problem with link congestion after about > 26 messages > > being sent. I'm running connectionless traffic ( also with > the socket > > set to non-blocking via fcntl), with the domain set to > closest first, > > the destination droppable flag set to FALSE, the message importance > > set to HIGH, and the message size set to 2000 bytes. > > Your MTU is probably 1500 B so TIPC will have to split each message > thereby sending 2 packets. After the 25th user-space send, you have 50 > items in the link queue (if you have a fast cpu and and ack message > from the remote tipc hasn't arrived > yet.) Since your link window is 50, the 26th send fails, and doesn't > block. > You retry getting EAGAIN until the far tipc node sends an ack, opening > up room in the link queue. Yes it really takes that long! If you > measure it it will likely only be ~100 us so long is a relative term. > > The only hitch is that you say you have set the message priority to > HIGH so the link congestion should be 50/3*5 as can be seen here: > <http://lxr.linux.no/source/net/tipc/link.c?v=2.6.17.13#L1037> > and here: > <http://lxr.linux.no/source/net/tipc/link.c?v=2.6.17.13#L2710> > > (tipc in 2.6.17.x is very close to tipc-1.5.12 and I don't have the > 1.5.12 code handy) > > > > > > When I run the program, which is a modified version of the > hello world > > program, with the server running on one node and the > client running > > on a different node I'm getting back the error: Resource > temporarily > > unavailable and an errno of 11(EAGAIN) after the 26th > message. So, I' > > updated my code to retry if an errno of EAGAIN is returned, > but I need > > to retry some value which was more than 2500 (I believe it > was around > > 2650 or something like > > that) times before it successfully can send 27 messages > over the link. > > Once the acks start coming back you should get fewer EAGAINs in a row > because of the way the protocol works... > > > > > > > > tipc-config -ls shows the following indicating that link > congestion > > is > > happening: > > > > Link <1.1.169:eth0-1.1.168:eth0> > > ACTIVE MTU:1500 Priority:10 Tolerance:1500 ms > Window:50 packets > > RX packets:29 fragments:0/0 bundles:0/0 > > TX packets:3627 fragments:3620/1809 bundles:0/0 > > TX profile sample:105 packets average:1356 octets > > 0-64:7% -256:0% -1024:40% -4096:53% -16354:0% -32768:0% -66000:0% > > RX states:2175142 probes:1087411 naks:0 defs:0 dups:0 > > TX states:2174945 probes:1087534 naks:0 acks:0 dups:0 > > Congestion bearer:0 link:36 Send queue max:36 avg:0 > > > > > > Any ideas as to why I'm seeing link congestion? > > Ummm, > You're code is sending too fast and you have asked to not block... > the network round trip time is longer than the time it takes to > enqueue 25 packets... ;-) > > > If you'd like I can attach some sample code. > > > > > > Just before sending this note, I tried commenting out the code that > > makes the socket non-blocking via fcntl and then I don't see link > > congestion anymore. So why does making the socket > non-blocking lead > > to link congestion? > > If blocking is allowed, the the calling process is put to sleep > briefly until the link congestion is abated: > http://lxr.linux.no/source/net/tipc/socket.c?v=2.6.17.13#L524 > > > So you have to keep trying to send, hopefully also doing other useful > work while you wait for the ack. > > Assuming that this is correct, then I'd like to suggest that tipc > could notify userspace that the link is no longer congested... I've > done things like that before but all in userspace. I'll explain at > length if anyone is interested in the details... > > Given that congestion abatement notification doesn't exist yet, you > have to: > 0. live with it, > 1. put in some flow control on the send side, 2. increase the link > send window, 3. both 1 & 2. > > Another question that comes to mind is: > What is the maximum time that a process be suspended? > I'd guess it would be the (default) link timeout time of ~1.5 s. > That would only occur if the far end failed. > -- > // Randy > > -------------------------------------------------------------- > ----------- > This SF.net email is sponsored by: Microsoft Defy all challenges. > Microsoft(R) Visual Studio 2005. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > tipc-discussion mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/tipc-discussion > ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ tipc-discussion mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/tipc-discussion
