Hi, Cesar: On Wed, 25 Nov 2009, Cesar Delgado wrote:
> > On Nov 25, 2009, at 2:05 PM, venugopal iyer wrote: > >> >> >> >> On Wed, 25 Nov 2009, Cesar Delgado wrote: >> >>> Venu, >>> >>> On Nov 25, 2009, at 9:49 AM, venugopal iyer wrote: >>> >>>> >>>> Hi, Cesar: >>>> >>>> On Tue, 24 Nov 2009, Cesar Delgado wrote: >>>> >>>>> Venugopal, >>>>> >>>>> I'm sorry if these sounds like basic questions. I really appreciate the >>>>> patience and the help. Replies in-line. >>>>> >>>>> On Nov 24, 2009, at 9:29 AM, venugopal iyer wrote: >>>>> >>>>>> >>>>>> >>>>>> Hi, Cesar: >>>>>> >>>>>> On Mon, 23 Nov 2009, Cesar Delgado wrote: >>>>>> >>>>>>> I'm setting up a server to go to a hosting site where I have a 1Mbps >>>>>>> pipe. From what I read I know I can't set the limit to this as the >>>>>>> lowest setting is ~1.2Mbps and this is something that's getting worked >>>>>>> on in Crossbow2. I am seeing some strange behavior. >>>>>>> >>>>>>> FIrst I have a question on flowadm's show-usage command. When I try to >>>>>>> run show-prop with the name of a flow I get an error. The flow exists. >>>>>>> What am I doing wrong? >>>>>>> >>>>>>> root at myhost:~# flowadm show-usage -f /var/log/net.log http-flow >>>>>>> flowadm: invalid flow: '(null)' >>>>>> >>>>>> This is a bug, I have submitted >>>>>> >>>>>> 6904427 flowadm show-usage doesn't work with a flow name >>>>> >>>>> Thanks for submitting that. I haven't been able to find a link to the >>>>> bugtracker for Crossbow. Could you please send me the URL? >>>> >>>> I think it should show up on >>>> http://bugs.opensolaris.org/bugdatabase/index.jsp soon. >>>> >>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>> Ok, now for my problem. I have the following setting: >>>>>>> >>>>>>> root at myhost:~# flowadm show-flowprop http-flow >>>>>>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>>>>>> http-flow maxbw 1.228 -- 1228k >>>>>>> http-flow priority medium -- medium >>>>>>> >>>>>>> I ran a test hitting the webserver and I see this: >>>>>> >>>>>> I have the following flow >>>>>> >>>>>> # flowadm show-flow FLOW LINK IPADDR >>>>>> PROTO LPORT RPORT DSFLD >>>>>> tcp-flow <link> -- tcp -- -- >>>>>> -- >>>>>> >>>>>> >>>>>> # flowadm show-flowprop tcp-flow >>>>>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>>>>> tcp-flow maxbw 1.228 -- 1228K >>>>>> tcp-flow priority -- -- ? >>>>>> >>>>>> When I send TCP traffic (am using a traffic generator - netperf, to >>>>>> this machine from a peer) for about 2 mins. >>>>>> >>>>>> On the peer the traffic generator (sender) says I am capped to about >>>>>> 1.14 Mbps. >>>>>> >>>>>> Size Size Size Time Throughput bytes bytes bytes >>>>>> secs. 10^6bits/sec >>>>>> >>>>>> 49152 49152 49152 120.49 1.14 >>>>>> >>>>>> Now, when I try show-usage during the traffic flow on >>>>>> the machine with the above flow in place (receiver), I am seeing: >>>>>> >>>>>> # flowadm show-usage -s 11/24/2009 -f /var/tmp/tcpflow >>>>>> FLOW START END RBYTES OBYTES BANDWIDTH >>>>>> tcp-flow 08:51:48 08:52:08 3428658 107802 1.414 Mbp >>>>>> tcp-flow 08:52:08 08:52:28 3431198 107802 1.415 Mbp >>>>>> tcp-flow 08:52:28 08:52:48 3434614 107888 1.417 Mbp >>>>>> tcp-flow 08:52:48 08:53:08 3443298 107802 1.420 Mbp >>>>>> tcp-flow 08:53:08 08:53:28 3444324 107802 1.420 Mbp >>>>>> tcp-flow 08:53:28 08:53:48 1376806 43576 0.568 Mbps >>>>>> >>>>>> ... >>>>>> >>>>>> I think the difference you see is likely to be because of the time >>>>>> period when the stats are written to the file (the bandwidth is computed >>>>>> for every 20 seconds period which might not be exactly in >>>>>> sync with the bandwidth enforcement period in the kernel) and also >>>>>> could be because of rounding up etc. But, if you look at the entire >>>>>> duration, it averages to about the configured limit (in the above >>>>>> example, I think it is about 1.275 Mbps for the 2 min duration) >>>>> >>>>> The way I'm testing it is setting up Apache and then moving down a file >>>>> with `wget`. The use case for this machine is an Apache based app that >>>>> serves large files to customers. That is why I think a `wget` is more >>>>> telling of "real" performance than netperf. I'm running the test again >>>>> and on the client side I am seeing usage over the maxbw limit I have set. >>>>> `wget` is reporting about 2Mbps transfer rate which is much closer to >>>>> what I was seeing in the show-usage statistics. >>>>> >>>> >>>>> [cdelgado at Bluegene tmp]$ wget sol/myfile.dat >>>>> --10:01:30-- http://sol/myfile.dat >>>>> Resolving sol... 192.168.69.104 >>>>> Connecting to sol|192.168.69.104|:80... connected. >>>>> HTTP request sent, awaiting response... 200 OK >>>>> Length: 1048576000 (1000M) >>>>> Saving to: `myfile.dat' >>>>> >>>>> 5% [==> ] 55,530,974 >>>>> 267K/s eta 60m 44s >>>>> >>>>> >>>>>> >>>>>> BTW, setting a maxbw for a link (dladm) doesn't really impact the >>>>>> flow as the bandwidth for both are independent. >>>>> >>>>> Thank you for this clarification but I still don't understand how I can >>>>> be seeing ~2Mbps transfer if both the link and the flow are both capped >>>>> at 1.2Mbps. >>>>> >>>> >>>> Can you try with a higher bandwidth, say 100 Mbps and see what the results >>>> are when compared to wget's output? >>>> >>>> Also, another way of manually checking would be to do a >>>> # kstat -c flow -n http-flow >>>> >>>> before and after the wget run and see how many bytes (rbytes) the >>>> kernel has seen for that flow (assuming there isn't any other traffic >>>> going over the flow), and then determine the bandwidth (you might need >>>> to get the duration of the wget run pretty close to get the >>>> right bandwdith value). >>>> >>>> -venu >>> >>> I changed the flow to be 100Mbps. >>> >>> root at myhost:/tmp# flowadm show-flowprop -p maxbw http-flow >>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>> http-flow maxbw 100 -- 100 >>> >>> I also removed the maxbw=1.228 I had set on the link. When I changed this >>> value on the link I lost all network to the machine. Had to go in and >>> manually reboot it. The network came up fine. >>> >>> I created two files one before wget and one after with no other network >>> traffic except the ssh session. I created a 1 GB file to transfer. The >>> output is a bit confusing to me. >>> >>> root at myhost:/tmp# ls http-flow-* >>> http-flow-1.txt http-flow-2.txt >>> root at myhost:/tmp# cat http-flow-* >>> module: unix instance: 0 >>> name: http-flow class: flow >>> crtime 188662.418688816 >>> ierrors 0 >>> ipackets 4680 >>> obytes 182520974 >>> oerrors 0 >>> opackets 127521 >>> rbytes 308138 >>> snaptime 189431.882363589 >>> >>> module: unix instance: 0 >>> name: http-flow class: flow >>> crtime 188662.418688816 >>> ierrors 0 >>> ipackets 31262 >>> obytes 1281785975 >>> oerrors 0 >>> opackets 895533 >>> rbytes 2062678 >>> snaptime 189595.122347726 >>> >>> So this means the network only passed ~1.6 MB of data? >> >> is this on the outbound side or inbound? >> >> looks like it has a total of 8794120008 bits in the outbound side (obytes) >> and 14036320 bits on the inbound (rbytes). What is the duration >> 93 seconds? So the outbound would be ~94 Mbps and inbound 150 Kbps. >> > > Sorry, outbound. I was looking at the wrong row. With the flow set at 50 > I'm getting getting ~37 Mbps. In my mind the flow is not respecting the > value I'm giving it. With the maxbw set to 100 I could get ~80 Mbps and now > I'm only seeing ~37Mbps. Could it be something in the way Crossbow is > averaging the actually bits and bytes going down the pipe or is it me not > understanding the system? For TCP you might get a little less than the set bandwidth as TCP would do flow control too (i.e if for bandwidth reasons ACKs come back slower or packets get dropped, TCP will react and will slow down). Also, note that the current property (maxbw) is a *limit* which means we will ensure that the flow will not get *more* than what you have asked for. Going forwards we are looking at adding guarantee where in the flow would get what it asked for. -venu > > root at myhost:/tmp# flowadm show-flowprop http-flow > FLOW PROPERTY VALUE DEFAULT POSSIBLE > http-flow maxbw 50 -- 50 > http-flow priority medium -- medium > > root at myhost:/tmp# cat http-flow-* > module: unix instance: 0 > name: http-flow class: flow > crtime 188662.418688816 > ierrors 0 > ipackets 32346 > obytes 1300129135 > oerrors 0 > opackets 908351 > rbytes 2133486 > snaptime 191261.607310645 > > module: unix instance: 0 > name: http-flow class: flow > crtime 188662.418688816 > ierrors 0 > ipackets 85361 > obytes 2399401106 > oerrors 0 > opackets 1676468 > rbytes 5632604 > snaptime 192621.873203347 > > > [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat > --13:52:20-- http://sol/myfile.dat > Resolving sol... 192.168.69.104 > Connecting to sol|192.168.69.104|:80... connected. > HTTP request sent, awaiting response... 200 OK > Length: 1048576000 (1000M) > Saving to: `myfile.dat' > > 100%[==========================================================>] > 1,048,576,000 4.06M/s in 3m 44s > > 13:56:04 (4.46 MB/s) - `myfile.dat' saved [1048576000/1048576000] > > > real 3m44.364s > user 0m1.042s > sys 0m24.065s > > > >> >>> >>> The wget command was telling me it was getting ~80Mbps which is under the >>> threshold of the flow. >>> >>> [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat >>> --13:21:49-- http://sol/myfile.dat >>> Resolving sol... 192.168.69.104 >>> Connecting to sol|192.168.69.104|:80... connected. >>> HTTP request sent, awaiting response... 200 OK >>> Length: 1048576000 (1000M) >>> Saving to: `myfile.dat' >>> >>> 100%[==========================================================>] >>> 1,048,576,000 10.5M/s in 93s >>> >>> 13:23:23 (10.7 MB/s) - `myfile.dat' saved [1048576000/1048576000] >>> >>> >>> real 1m33.701s >>> user 0m0.899s >>> sys 0m23.874s >>> >>> Maybe a value of 100Mbps is too high for this machine. I might try with >>> 50Mbps to see what wget says. >>> >> >> OK, >> >> -venu >> >>> -Cesar >>> >>> >>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> >>>>>>> root at myhost:~# flowadm show-usage -s 11/23/2009,01:32:22 -e >>>>>>> 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW" >>>>>>> http-flow 01:32:22 01:32:42 1512 2571 0.001 Mbps >>>>>>> ssh-flow 01:32:42 01:33:02 1818 3578 0.002 Mbps >>>>>>> http-flow 01:33:02 01:33:22 66917 3165136 1.292 Mbp >>>>>>> ssh-flow 01:33:02 01:33:22 3618 5344 0.003 Mbps >>>>>>> http-flow 01:33:22 01:33:42 117947 5713018 2.332 Mbp >>>>>>> ssh-flow 01:33:22 01:33:42 4182 3020 0.002 Mbps >>>>>>> http-flow 01:33:42 01:34:02 118998 5685520 2.321 Mbp >>>>>>> ssh-flow 01:33:42 01:34:02 11616 9924 0.008 Mbps >>>>>>> http-flow 01:34:02 01:34:22 117084 5725664 2.337 Mbp >>>>>>> http-flow 01:34:22 01:34:42 119130 5725168 2.337 Mbp >>>>>>> http-flow 01:34:42 01:35:02 114180 5725168 2.335 Mbp >>>>>>> http-flow 01:35:02 01:35:22 109230 5725664 2.333 Mbp >>>>>>> http-flow 01:35:22 01:35:42 116160 5725168 2.336 Mbp >>>>>>> http-flow 01:35:42 01:36:02 119262 5725168 2.337 Mbp >>>>>>> http-flow 01:36:02 01:36:22 119196 5725664 2.337 Mbp >>>>>>> http-flow 01:36:22 01:36:42 117216 5725168 2.336 Mbp >>>>>>> http-flow 01:36:42 01:37:02 119394 5722636 2.336 Mbp >>>>>>> http-flow 01:37:02 01:37:22 119526 5725168 2.337 Mbp >>>>>>> http-flow 01:37:22 01:37:42 119460 5725168 2.337 Mbp >>>>>>> http-flow 01:37:42 01:38:02 119460 5725664 2.338 Mbp >>>>>>> http-flow 01:38:02 01:38:22 119724 5725168 2.337 Mbp >>>>>>> http-flow 01:38:22 01:38:42 119724 5725168 2.337 Mbp >>>>>>> http-flow 01:38:42 01:39:02 119130 5722636 2.336 Mbp >>>>>>> http-flow 01:39:02 01:39:22 118866 5725168 2.337 Mbp >>>>>>> http-flow 01:39:22 01:39:42 116490 5725664 2.336 Mbp >>>>>>> http-flow 01:39:42 01:40:02 119790 5725168 2.337 Mbp >>>>>>> http-flow 01:40:02 01:40:22 117678 5725168 2.337 Mbp >>>>>>> http-flow 01:40:22 01:40:42 118668 5725664 2.337 Mbp >>>>>>> http-flow 01:40:42 01:41:02 117414 5725168 2.337 Mbp >>>>>>> http-flow 01:41:02 01:41:22 119790 5725168 2.337 Mbp >>>>>>> http-flow 01:41:22 01:41:42 119813 5720510 2.336 Mbp >>>>>>> http-flow 01:41:42 01:42:02 119394 5725664 2.338 Mbp >>>>>>> http-flow 01:42:02 01:42:22 119724 5722272 2.336 Mbp >>>>>>> http-flow 01:42:22 01:42:42 119526 5725664 2.338 Mbp >>>>>>> http-flow 01:42:42 01:43:02 119196 5722140 2.336 Mbp >>>>>>> http-flow 01:43:02 01:43:22 119394 5725664 2.338 Mbp >>>>>>> http-flow 01:43:22 01:43:42 119658 5725168 2.337 Mbp >>>>>>> http-flow 01:43:42 01:44:02 119064 5725168 2.337 Mbp >>>>>>> http-flow 01:44:02 01:44:22 113256 5676668 2.315 Mbp >>>>>>> ssh-flow 01:44:02 01:44:22 18414 49646 0.027 Mbps >>>>>>> http-flow 01:44:22 01:44:42 118206 5725664 2.337 Mbp >>>>>>> http-flow 01:44:42 01:45:02 117282 5722140 2.335 Mbp >>>>>>> ssh-flow 01:44:42 01:45:02 4698 3544 0.003 Mbps >>>>>>> http-flow 01:45:02 01:45:22 118536 5688284 2.322 Mbp >>>>>>> ssh-flow 01:45:02 01:45:22 4092 3198 0.002 Mbps >>>>>>> http-flow 01:45:22 01:45:42 119130 5725168 2.337 Mbp >>>>>>> ssh-flow 01:45:22 01:45:42 1980 1478 0.001 Mbps >>>>>>> >>>>>>> That's above the flow's maxbw parameter. After that I tried to change >>>>>>> the maxbw of the link with dladm and that brought the bandwidth down >>>>>>> but still not down to 1.2 Mbps. >>>>>>> >>>>>>> root at myhost:~# dladm show-linkprop -p maxbw e1000g0 >>>>>>> LINK PROPERTY PERM VALUE DEFAULT POSSIBLE >>>>>>> e1000g0 maxbw rw 1.228 -- -- >>>>>>> >>>>>>> root at myhost:~# flowadm show-usage -s 11/23/2009,01:46:02 -e >>>>>>> 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW" >>>>>>> http-flow 01:46:02 01:46:22 119394 5725168 2.337 Mbp >>>>>>> ssh-flow 01:46:02 01:46:22 4680 2980 0.003 Mbps >>>>>>> http-flow 01:46:22 01:46:42 94314 4520316 1.845 Mbp >>>>>>> >>>>>>> Any ideas or is there a subtlety that I'm missing and the behavior is >>>>>>> correct? >>>>>>> >>>>>>> Thanks for the help. >>>>>>> >>>>>>> -Cesar >>>>>>> >>>>>>> _______________________________________________ >>>>>>> crossbow-discuss mailing list >>>>>>> crossbow-discuss at opensolaris.org >>>>>>> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss >>>>>>> >>>>> >>>>> >>> >>> > >
