Cesar:
On Thu, 26 Nov 2009, Cesar Delgado wrote: > > > On Nov 25, 2009, at 3:39 PM, venugopal iyer wrote: > >> >> Hi, Cesar: >> >> On Wed, 25 Nov 2009, Cesar Delgado wrote: >> >>> >>> On Nov 25, 2009, at 2:05 PM, venugopal iyer wrote: >>> >>>> >>>> >>>> >>>> On Wed, 25 Nov 2009, Cesar Delgado wrote: >>>> >>>>> Venu, >>>>> >>>>> On Nov 25, 2009, at 9:49 AM, venugopal iyer wrote: >>>>> >>>>>> >>>>>> Hi, Cesar: >>>>>> >>>>>> On Tue, 24 Nov 2009, Cesar Delgado wrote: >>>>>> >>>>>>> Venugopal, >>>>>>> >>>>>>> I'm sorry if these sounds like basic questions. I really appreciate >>>>>>> the patience and the help. Replies in-line. >>>>>>> >>>>>>> On Nov 24, 2009, at 9:29 AM, venugopal iyer wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Hi, Cesar: >>>>>>>> >>>>>>>> On Mon, 23 Nov 2009, Cesar Delgado wrote: >>>>>>>> >>>>>>>>> I'm setting up a server to go to a hosting site where I have a 1Mbps >>>>>>>>> pipe. From what I read I know I can't set the limit to this as the >>>>>>>>> lowest setting is ~1.2Mbps and this is something that's getting >>>>>>>>> worked on in Crossbow2. I am seeing some strange behavior. >>>>>>>>> >>>>>>>>> FIrst I have a question on flowadm's show-usage command. When I try >>>>>>>>> to run show-prop with the name of a flow I get an error. The flow >>>>>>>>> exists. What am I doing wrong? >>>>>>>>> >>>>>>>>> root at myhost:~# flowadm show-usage -f /var/log/net.log http-flow >>>>>>>>> flowadm: invalid flow: '(null)' >>>>>>>> >>>>>>>> This is a bug, I have submitted >>>>>>>> >>>>>>>> 6904427 flowadm show-usage doesn't work with a flow name >>>>>>> >>>>>>> Thanks for submitting that. I haven't been able to find a link to the >>>>>>> bugtracker for Crossbow. Could you please send me the URL? >>>>>> >>>>>> I think it should show up on >>>>>> http://bugs.opensolaris.org/bugdatabase/index.jsp soon. >>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Ok, now for my problem. I have the following setting: >>>>>>>>> >>>>>>>>> root at myhost:~# flowadm show-flowprop http-flow >>>>>>>>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>>>>>>>> http-flow maxbw 1.228 -- 1228k >>>>>>>>> http-flow priority medium -- medium >>>>>>>>> >>>>>>>>> I ran a test hitting the webserver and I see this: >>>>>>>> >>>>>>>> I have the following flow >>>>>>>> >>>>>>>> # flowadm show-flow FLOW LINK IPADDR >>>>>>>> PROTO LPORT RPORT DSFLD >>>>>>>> tcp-flow <link> -- tcp -- -- >>>>>>>> -- >>>>>>>> >>>>>>>> >>>>>>>> # flowadm show-flowprop tcp-flow >>>>>>>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>>>>>>> tcp-flow maxbw 1.228 -- 1228K >>>>>>>> tcp-flow priority -- -- ? >>>>>>>> >>>>>>>> When I send TCP traffic (am using a traffic generator - netperf, to >>>>>>>> this machine from a peer) for about 2 mins. >>>>>>>> >>>>>>>> On the peer the traffic generator (sender) says I am capped to about >>>>>>>> 1.14 Mbps. >>>>>>>> >>>>>>>> Size Size Size Time Throughput bytes bytes bytes >>>>>>>> secs. 10^6bits/sec >>>>>>>> >>>>>>>> 49152 49152 49152 120.49 1.14 >>>>>>>> >>>>>>>> Now, when I try show-usage during the traffic flow on >>>>>>>> the machine with the above flow in place (receiver), I am seeing: >>>>>>>> >>>>>>>> # flowadm show-usage -s 11/24/2009 -f /var/tmp/tcpflow >>>>>>>> FLOW START END RBYTES OBYTES BANDWIDTH >>>>>>>> tcp-flow 08:51:48 08:52:08 3428658 107802 1.414 >>>>>>>> Mbp >>>>>>>> tcp-flow 08:52:08 08:52:28 3431198 107802 1.415 >>>>>>>> Mbp >>>>>>>> tcp-flow 08:52:28 08:52:48 3434614 107888 1.417 >>>>>>>> Mbp >>>>>>>> tcp-flow 08:52:48 08:53:08 3443298 107802 1.420 >>>>>>>> Mbp >>>>>>>> tcp-flow 08:53:08 08:53:28 3444324 107802 1.420 >>>>>>>> Mbp >>>>>>>> tcp-flow 08:53:28 08:53:48 1376806 43576 0.568 Mbps >>>>>>>> >>>>>>>> ... >>>>>>>> >>>>>>>> I think the difference you see is likely to be because of the time >>>>>>>> period when the stats are written to the file (the bandwidth is >>>>>>>> computed for every 20 seconds period which might not be exactly in >>>>>>>> sync with the bandwidth enforcement period in the kernel) and also >>>>>>>> could be because of rounding up etc. But, if you look at the entire >>>>>>>> duration, it averages to about the configured limit (in the above >>>>>>>> example, I think it is about 1.275 Mbps for the 2 min duration) >>>>>>> >>>>>>> The way I'm testing it is setting up Apache and then moving down a file >>>>>>> with `wget`. The use case for this machine is an Apache based app that >>>>>>> serves large files to customers. That is why I think a `wget` is more >>>>>>> telling of "real" performance than netperf. I'm running the test again >>>>>>> and on the client side I am seeing usage over the maxbw limit I have >>>>>>> set. `wget` is reporting about 2Mbps transfer rate which is much >>>>>>> closer to what I was seeing in the show-usage statistics. >>>>>>> >>>>>> >>>>>>> [cdelgado at Bluegene tmp]$ wget sol/myfile.dat >>>>>>> --10:01:30-- http://sol/myfile.dat >>>>>>> Resolving sol... 192.168.69.104 >>>>>>> Connecting to sol|192.168.69.104|:80... connected. >>>>>>> HTTP request sent, awaiting response... 200 OK >>>>>>> Length: 1048576000 (1000M) >>>>>>> Saving to: `myfile.dat' >>>>>>> >>>>>>> 5% [==> ] >>>>>>> 55,530,974 267K/s eta 60m 44s >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> BTW, setting a maxbw for a link (dladm) doesn't really impact the >>>>>>>> flow as the bandwidth for both are independent. >>>>>>> >>>>>>> Thank you for this clarification but I still don't understand how I can >>>>>>> be seeing ~2Mbps transfer if both the link and the flow are both capped >>>>>>> at 1.2Mbps. >>>>>>> >>>>>> >>>>>> Can you try with a higher bandwidth, say 100 Mbps and see what the >>>>>> results >>>>>> are when compared to wget's output? >>>>>> >>>>>> Also, another way of manually checking would be to do a >>>>>> # kstat -c flow -n http-flow >>>>>> >>>>>> before and after the wget run and see how many bytes (rbytes) the >>>>>> kernel has seen for that flow (assuming there isn't any other traffic >>>>>> going over the flow), and then determine the bandwidth (you might need >>>>>> to get the duration of the wget run pretty close to get the >>>>>> right bandwdith value). >>>>>> >>>>>> -venu >>>>> >>>>> I changed the flow to be 100Mbps. >>>>> >>>>> root at myhost:/tmp# flowadm show-flowprop -p maxbw http-flow >>>>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>>>> http-flow maxbw 100 -- 100 >>>>> >>>>> I also removed the maxbw=1.228 I had set on the link. When I changed >>>>> this value on the link I lost all network to the machine. Had to go in >>>>> and manually reboot it. The network came up fine. >>>>> >>>>> I created two files one before wget and one after with no other network >>>>> traffic except the ssh session. I created a 1 GB file to transfer. The >>>>> output is a bit confusing to me. >>>>> >>>>> root at myhost:/tmp# ls http-flow-* >>>>> http-flow-1.txt http-flow-2.txt >>>>> root at myhost:/tmp# cat http-flow-* >>>>> module: unix instance: 0 >>>>> name: http-flow class: flow >>>>> crtime 188662.418688816 >>>>> ierrors 0 >>>>> ipackets 4680 >>>>> obytes 182520974 >>>>> oerrors 0 >>>>> opackets 127521 >>>>> rbytes 308138 >>>>> snaptime 189431.882363589 >>>>> >>>>> module: unix instance: 0 >>>>> name: http-flow class: flow >>>>> crtime 188662.418688816 >>>>> ierrors 0 >>>>> ipackets 31262 >>>>> obytes 1281785975 >>>>> oerrors 0 >>>>> opackets 895533 >>>>> rbytes 2062678 >>>>> snaptime 189595.122347726 >>>>> >>>>> So this means the network only passed ~1.6 MB of data? >>>> >>>> is this on the outbound side or inbound? >>>> >>>> looks like it has a total of 8794120008 bits in the outbound side (obytes) >>>> and 14036320 bits on the inbound (rbytes). What is the duration >>>> 93 seconds? So the outbound would be ~94 Mbps and inbound 150 Kbps. >>>> >>> >>> Sorry, outbound. I was looking at the wrong row. With the flow set at 50 >>> I'm getting getting ~37 Mbps. In my mind the flow is not respecting the >>> value I'm giving it. With the maxbw set to 100 I could get ~80 Mbps and >>> now I'm only seeing ~37Mbps. Could it be something in the way Crossbow is >>> averaging the actually bits and bytes going down the pipe or is it me not >>> understanding the system? >> >> For TCP you might get a little less than the set bandwidth as TCP >> would do flow control too (i.e if for bandwidth reasons ACKs come back >> slower or packets get dropped, TCP will react and will slow down). Also, >> note that the current property (maxbw) is a *limit* which means we will >> ensure that the flow will not get *more* than what you have asked for. >> Going forwards we are looking at adding guarantee where in the flow would >> get what it asked for. >> >> -venu > > Good to know. And it's true about TCP, for some reason I though it wouldn't > happen with maxbw but I do unserstand now t hat the entire TVP connection is > being affected by the parameter, including all the extra TCP stuff that's > needed like the handshake. I don't need t set guarantees at the moment so > this doesn't worry me much. I just want to cap the usage so I don't go over > my 1Mbit network quota. Like overages in cellphones, network overages are > expensive. > > So now back to my original problem. In the new test I'm seeing ~2.1Mbps, > that's almost double the maxbw value. If maxbw is a limit as you say in the > previous reply, why am I seeing more than 1.2Mbps when I set maxbw to 1228k? > Could it be a corner case being that it's the smallest value possible? Looking at the output below, I suspect so. Let me try it at my end and check this.. will let you know. -venu > > cdelgado at myhost:~$ tail /tmp/http-flow-* > ==> /tmp/http-flow-1.txt <== > name: http-flow class: flow > crtime 188662.418688816 > ierrors 0 > ipackets 101961 > obytes 2446610048 > oerrors 0 > opackets 1709453 > rbytes 6727462 > snaptime 198359.572083418 > > > ==> /tmp/http-flow-2.txt <== > name: http-flow class: flow > crtime 188662.418688816 > ierrors 0 > ipackets 485927 > obytes 3545874452 > oerrors 0 > opackets 2477456 > rbytes 32069346 > snaptime 202630.597287968 > > > [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat > --15:50:35-- http://sol/myfile.dat > Resolving sol... 192.168.69.104 > Connecting to sol|192.168.69.104|:80... connected. > HTTP request sent, awaiting response... 200 OK > Length: 1048576000 (1000M) > Saving to: `myfile.dat' > > 100%[==========================================================>] > 1,048,576,000 267K/s in 64m 8s > > 16:54:43 (266 KB/s) - `myfile.dat' saved [1048576000/1048576000] > > > real 64m7.855s > user 0m4.982s > sys 0m32.007s > > >> >>> >>> root at myhost:/tmp# flowadm show-flowprop http-flow >>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>> http-flow maxbw 50 -- 50 >>> http-flow priority medium -- medium >>> >>> root at myhost:/tmp# cat http-flow-* >>> module: unix instance: 0 >>> name: http-flow class: flow >>> crtime 188662.418688816 >>> ierrors 0 >>> ipackets 32346 >>> obytes 1300129135 >>> oerrors 0 >>> opackets 908351 >>> rbytes 2133486 >>> snaptime 191261.607310645 >>> >>> module: unix instance: 0 >>> name: http-flow class: flow >>> crtime 188662.418688816 >>> ierrors 0 >>> ipackets 85361 >>> obytes 2399401106 >>> oerrors 0 >>> opackets 1676468 >>> rbytes 5632604 >>> snaptime 192621.873203347 >>> >>> >>> [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat >>> --13:52:20-- http://sol/myfile.dat >>> Resolving sol... 192.168.69.104 >>> Connecting to sol|192.168.69.104|:80... connected. >>> HTTP request sent, awaiting response... 200 OK >>> Length: 1048576000 (1000M) >>> Saving to: `myfile.dat' >>> >>> 100%[==========================================================>] >>> 1,048,576,000 4.06M/s in 3m 44s >>> >>> 13:56:04 (4.46 MB/s) - `myfile.dat' saved [1048576000/1048576000] >>> >>> >>> real 3m44.364s >>> user 0m1.042s >>> sys 0m24.065s >>> >>> >>> >>>> >>>>> >>>>> The wget command was telling me it was getting ~80Mbps which is under the >>>>> threshold of the flow. >>>>> >>>>> [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat >>>>> --13:21:49-- http://sol/myfile.dat >>>>> Resolving sol... 192.168.69.104 >>>>> Connecting to sol|192.168.69.104|:80... connected. >>>>> HTTP request sent, awaiting response... 200 OK >>>>> Length: 1048576000 (1000M) >>>>> Saving to: `myfile.dat' >>>>> >>>>> 100%[==========================================================>] >>>>> 1,048,576,000 10.5M/s in 93s >>>>> >>>>> 13:23:23 (10.7 MB/s) - `myfile.dat' saved [1048576000/1048576000] >>>>> >>>>> >>>>> real 1m33.701s >>>>> user 0m0.899s >>>>> sys 0m23.874s >>>>> >>>>> Maybe a value of 100Mbps is too high for this machine. I might try with >>>>> 50Mbps to see what wget says. >>>>> >>>> >>>> OK, >>>> >>>> -venu >>>> >>>>> -Cesar >>>>> >>>>> >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> root at myhost:~# flowadm show-usage -s 11/23/2009,01:32:22 -e >>>>>>>>> 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW" >>>>>>>>> http-flow 01:32:22 01:32:42 1512 2571 0.001 Mbps >>>>>>>>> ssh-flow 01:32:42 01:33:02 1818 3578 0.002 Mbps >>>>>>>>> http-flow 01:33:02 01:33:22 66917 3165136 1.292 >>>>>>>>> Mbp >>>>>>>>> ssh-flow 01:33:02 01:33:22 3618 5344 0.003 Mbps >>>>>>>>> http-flow 01:33:22 01:33:42 117947 5713018 2.332 >>>>>>>>> Mbp >>>>>>>>> ssh-flow 01:33:22 01:33:42 4182 3020 0.002 Mbps >>>>>>>>> http-flow 01:33:42 01:34:02 118998 5685520 2.321 >>>>>>>>> Mbp >>>>>>>>> ssh-flow 01:33:42 01:34:02 11616 9924 0.008 Mbps >>>>>>>>> http-flow 01:34:02 01:34:22 117084 5725664 2.337 >>>>>>>>> Mbp >>>>>>>>> http-flow 01:34:22 01:34:42 119130 5725168 2.337 >>>>>>>>> Mbp >>>>>>>>> http-flow 01:34:42 01:35:02 114180 5725168 2.335 >>>>>>>>> Mbp >>>>>>>>> http-flow 01:35:02 01:35:22 109230 5725664 2.333 >>>>>>>>> Mbp >>>>>>>>> http-flow 01:35:22 01:35:42 116160 5725168 2.336 >>>>>>>>> Mbp >>>>>>>>> http-flow 01:35:42 01:36:02 119262 5725168 2.337 >>>>>>>>> Mbp >>>>>>>>> http-flow 01:36:02 01:36:22 119196 5725664 2.337 >>>>>>>>> Mbp >>>>>>>>> http-flow 01:36:22 01:36:42 117216 5725168 2.336 >>>>>>>>> Mbp >>>>>>>>> http-flow 01:36:42 01:37:02 119394 5722636 2.336 >>>>>>>>> Mbp >>>>>>>>> http-flow 01:37:02 01:37:22 119526 5725168 2.337 >>>>>>>>> Mbp >>>>>>>>> http-flow 01:37:22 01:37:42 119460 5725168 2.337 >>>>>>>>> Mbp >>>>>>>>> http-flow 01:37:42 01:38:02 119460 5725664 2.338 >>>>>>>>> Mbp >>>>>>>>> http-flow 01:38:02 01:38:22 119724 5725168 2.337 >>>>>>>>> Mbp >>>>>>>>> http-flow 01:38:22 01:38:42 119724 5725168 2.337 >>>>>>>>> Mbp >>>>>>>>> http-flow 01:38:42 01:39:02 119130 5722636 2.336 >>>>>>>>> Mbp >>>>>>>>> http-flow 01:39:02 01:39:22 118866 5725168 2.337 >>>>>>>>> Mbp >>>>>>>>> http-flow 01:39:22 01:39:42 116490 5725664 2.336 >>>>>>>>> Mbp >>>>>>>>> http-flow 01:39:42 01:40:02 119790 5725168 2.337 >>>>>>>>> Mbp >>>>>>>>> http-flow 01:40:02 01:40:22 117678 5725168 2.337 >>>>>>>>> Mbp >>>>>>>>> http-flow 01:40:22 01:40:42 118668 5725664 2.337 >>>>>>>>> Mbp >>>>>>>>> http-flow 01:40:42 01:41:02 117414 5725168 2.337 >>>>>>>>> Mbp >>>>>>>>> http-flow 01:41:02 01:41:22 119790 5725168 2.337 >>>>>>>>> Mbp >>>>>>>>> http-flow 01:41:22 01:41:42 119813 5720510 2.336 >>>>>>>>> Mbp >>>>>>>>> http-flow 01:41:42 01:42:02 119394 5725664 2.338 >>>>>>>>> Mbp >>>>>>>>> http-flow 01:42:02 01:42:22 119724 5722272 2.336 >>>>>>>>> Mbp >>>>>>>>> http-flow 01:42:22 01:42:42 119526 5725664 2.338 >>>>>>>>> Mbp >>>>>>>>> http-flow 01:42:42 01:43:02 119196 5722140 2.336 >>>>>>>>> Mbp >>>>>>>>> http-flow 01:43:02 01:43:22 119394 5725664 2.338 >>>>>>>>> Mbp >>>>>>>>> http-flow 01:43:22 01:43:42 119658 5725168 2.337 >>>>>>>>> Mbp >>>>>>>>> http-flow 01:43:42 01:44:02 119064 5725168 2.337 >>>>>>>>> Mbp >>>>>>>>> http-flow 01:44:02 01:44:22 113256 5676668 2.315 >>>>>>>>> Mbp >>>>>>>>> ssh-flow 01:44:02 01:44:22 18414 49646 0.027 Mbps >>>>>>>>> http-flow 01:44:22 01:44:42 118206 5725664 2.337 >>>>>>>>> Mbp >>>>>>>>> http-flow 01:44:42 01:45:02 117282 5722140 2.335 >>>>>>>>> Mbp >>>>>>>>> ssh-flow 01:44:42 01:45:02 4698 3544 0.003 Mbps >>>>>>>>> http-flow 01:45:02 01:45:22 118536 5688284 2.322 >>>>>>>>> Mbp >>>>>>>>> ssh-flow 01:45:02 01:45:22 4092 3198 0.002 Mbps >>>>>>>>> http-flow 01:45:22 01:45:42 119130 5725168 2.337 >>>>>>>>> Mbp >>>>>>>>> ssh-flow 01:45:22 01:45:42 1980 1478 0.001 Mbps >>>>>>>>> >>>>>>>>> That's above the flow's maxbw parameter. After that I tried to >>>>>>>>> change the maxbw of the link with dladm and that brought the >>>>>>>>> bandwidth down but still not down to 1.2 Mbps. >>>>>>>>> >>>>>>>>> root at myhost:~# dladm show-linkprop -p maxbw e1000g0 >>>>>>>>> LINK PROPERTY PERM VALUE DEFAULT >>>>>>>>> POSSIBLE >>>>>>>>> e1000g0 maxbw rw 1.228 -- -- >>>>>>>>> >>>>>>>>> root at myhost:~# flowadm show-usage -s 11/23/2009,01:46:02 -e >>>>>>>>> 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW" >>>>>>>>> http-flow 01:46:02 01:46:22 119394 5725168 2.337 >>>>>>>>> Mbp >>>>>>>>> ssh-flow 01:46:02 01:46:22 4680 2980 0.003 Mbps >>>>>>>>> http-flow 01:46:22 01:46:42 94314 4520316 1.845 >>>>>>>>> Mbp >>>>>>>>> >>>>>>>>> Any ideas or is there a subtlety that I'm missing and the behavior is >>>>>>>>> correct? >>>>>>>>> >>>>>>>>> Thanks for the help. >>>>>>>>> >>>>>>>>> -Cesar >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> crossbow-discuss mailing list >>>>>>>>> crossbow-discuss at opensolaris.org >>>>>>>>> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss >>>>>>>>> >>>>>>> >>>>>>> >>>>> >>>>> >>> >>> > >
