Venu, Any new information on this? Is it a bug?
-Cesar On Nov 25, 2009, at 3:39 PM, venugopal iyer wrote: > > Hi, Cesar: > > On Wed, 25 Nov 2009, Cesar Delgado wrote: > >> >> On Nov 25, 2009, at 2:05 PM, venugopal iyer wrote: >> >>> >>> >>> >>> On Wed, 25 Nov 2009, Cesar Delgado wrote: >>> >>>> Venu, >>>> >>>> On Nov 25, 2009, at 9:49 AM, venugopal iyer wrote: >>>> >>>>> >>>>> Hi, Cesar: >>>>> >>>>> On Tue, 24 Nov 2009, Cesar Delgado wrote: >>>>> >>>>>> Venugopal, >>>>>> >>>>>> I'm sorry if these sounds like basic questions. I really appreciate the >>>>>> patience and the help. Replies in-line. >>>>>> >>>>>> On Nov 24, 2009, at 9:29 AM, venugopal iyer wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> Hi, Cesar: >>>>>>> >>>>>>> On Mon, 23 Nov 2009, Cesar Delgado wrote: >>>>>>> >>>>>>>> I'm setting up a server to go to a hosting site where I have a 1Mbps >>>>>>>> pipe. From what I read I know I can't set the limit to this as the >>>>>>>> lowest setting is ~1.2Mbps and this is something that's getting worked >>>>>>>> on in Crossbow2. I am seeing some strange behavior. >>>>>>>> >>>>>>>> FIrst I have a question on flowadm's show-usage command. When I try >>>>>>>> to run show-prop with the name of a flow I get an error. The flow >>>>>>>> exists. What am I doing wrong? >>>>>>>> >>>>>>>> root at myhost:~# flowadm show-usage -f /var/log/net.log http-flow >>>>>>>> flowadm: invalid flow: '(null)' >>>>>>> >>>>>>> This is a bug, I have submitted >>>>>>> >>>>>>> 6904427 flowadm show-usage doesn't work with a flow name >>>>>> >>>>>> Thanks for submitting that. I haven't been able to find a link to the >>>>>> bugtracker for Crossbow. Could you please send me the URL? >>>>> >>>>> I think it should show up on >>>>> http://bugs.opensolaris.org/bugdatabase/index.jsp soon. >>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Ok, now for my problem. I have the following setting: >>>>>>>> >>>>>>>> root at myhost:~# flowadm show-flowprop http-flow >>>>>>>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>>>>>>> http-flow maxbw 1.228 -- 1228k >>>>>>>> http-flow priority medium -- medium >>>>>>>> >>>>>>>> I ran a test hitting the webserver and I see this: >>>>>>> >>>>>>> I have the following flow >>>>>>> >>>>>>> # flowadm show-flow FLOW LINK IPADDR >>>>>>> PROTO LPORT RPORT DSFLD >>>>>>> tcp-flow <link> -- tcp -- -- >>>>>>> -- >>>>>>> >>>>>>> >>>>>>> # flowadm show-flowprop tcp-flow >>>>>>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>>>>>> tcp-flow maxbw 1.228 -- 1228K >>>>>>> tcp-flow priority -- -- ? >>>>>>> >>>>>>> When I send TCP traffic (am using a traffic generator - netperf, to >>>>>>> this machine from a peer) for about 2 mins. >>>>>>> >>>>>>> On the peer the traffic generator (sender) says I am capped to about >>>>>>> 1.14 Mbps. >>>>>>> >>>>>>> Size Size Size Time Throughput bytes bytes bytes >>>>>>> secs. 10^6bits/sec >>>>>>> >>>>>>> 49152 49152 49152 120.49 1.14 >>>>>>> >>>>>>> Now, when I try show-usage during the traffic flow on >>>>>>> the machine with the above flow in place (receiver), I am seeing: >>>>>>> >>>>>>> # flowadm show-usage -s 11/24/2009 -f /var/tmp/tcpflow >>>>>>> FLOW START END RBYTES OBYTES BANDWIDTH >>>>>>> tcp-flow 08:51:48 08:52:08 3428658 107802 1.414 Mbp >>>>>>> tcp-flow 08:52:08 08:52:28 3431198 107802 1.415 Mbp >>>>>>> tcp-flow 08:52:28 08:52:48 3434614 107888 1.417 Mbp >>>>>>> tcp-flow 08:52:48 08:53:08 3443298 107802 1.420 Mbp >>>>>>> tcp-flow 08:53:08 08:53:28 3444324 107802 1.420 Mbp >>>>>>> tcp-flow 08:53:28 08:53:48 1376806 43576 0.568 Mbps >>>>>>> >>>>>>> ... >>>>>>> >>>>>>> I think the difference you see is likely to be because of the time >>>>>>> period when the stats are written to the file (the bandwidth is >>>>>>> computed for every 20 seconds period which might not be exactly in >>>>>>> sync with the bandwidth enforcement period in the kernel) and also >>>>>>> could be because of rounding up etc. But, if you look at the entire >>>>>>> duration, it averages to about the configured limit (in the above >>>>>>> example, I think it is about 1.275 Mbps for the 2 min duration) >>>>>> >>>>>> The way I'm testing it is setting up Apache and then moving down a file >>>>>> with `wget`. The use case for this machine is an Apache based app that >>>>>> serves large files to customers. That is why I think a `wget` is more >>>>>> telling of "real" performance than netperf. I'm running the test again >>>>>> and on the client side I am seeing usage over the maxbw limit I have >>>>>> set. `wget` is reporting about 2Mbps transfer rate which is much closer >>>>>> to what I was seeing in the show-usage statistics. >>>>>> >>>>> >>>>>> [cdelgado at Bluegene tmp]$ wget sol/myfile.dat >>>>>> --10:01:30-- http://sol/myfile.dat >>>>>> Resolving sol... 192.168.69.104 >>>>>> Connecting to sol|192.168.69.104|:80... connected. >>>>>> HTTP request sent, awaiting response... 200 OK >>>>>> Length: 1048576000 (1000M) >>>>>> Saving to: `myfile.dat' >>>>>> >>>>>> 5% [==> ] >>>>>> 55,530,974 267K/s eta 60m 44s >>>>>> >>>>>> >>>>>>> >>>>>>> BTW, setting a maxbw for a link (dladm) doesn't really impact the >>>>>>> flow as the bandwidth for both are independent. >>>>>> >>>>>> Thank you for this clarification but I still don't understand how I can >>>>>> be seeing ~2Mbps transfer if both the link and the flow are both capped >>>>>> at 1.2Mbps. >>>>>> >>>>> >>>>> Can you try with a higher bandwidth, say 100 Mbps and see what the results >>>>> are when compared to wget's output? >>>>> >>>>> Also, another way of manually checking would be to do a >>>>> # kstat -c flow -n http-flow >>>>> >>>>> before and after the wget run and see how many bytes (rbytes) the >>>>> kernel has seen for that flow (assuming there isn't any other traffic >>>>> going over the flow), and then determine the bandwidth (you might need >>>>> to get the duration of the wget run pretty close to get the >>>>> right bandwdith value). >>>>> >>>>> -venu >>>> >>>> I changed the flow to be 100Mbps. >>>> >>>> root at myhost:/tmp# flowadm show-flowprop -p maxbw http-flow >>>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>>> http-flow maxbw 100 -- 100 >>>> >>>> I also removed the maxbw=1.228 I had set on the link. When I changed this >>>> value on the link I lost all network to the machine. Had to go in and >>>> manually reboot it. The network came up fine. >>>> >>>> I created two files one before wget and one after with no other network >>>> traffic except the ssh session. I created a 1 GB file to transfer. The >>>> output is a bit confusing to me. >>>> >>>> root at myhost:/tmp# ls http-flow-* >>>> http-flow-1.txt http-flow-2.txt >>>> root at myhost:/tmp# cat http-flow-* >>>> module: unix instance: 0 >>>> name: http-flow class: flow >>>> crtime 188662.418688816 >>>> ierrors 0 >>>> ipackets 4680 >>>> obytes 182520974 >>>> oerrors 0 >>>> opackets 127521 >>>> rbytes 308138 >>>> snaptime 189431.882363589 >>>> >>>> module: unix instance: 0 >>>> name: http-flow class: flow >>>> crtime 188662.418688816 >>>> ierrors 0 >>>> ipackets 31262 >>>> obytes 1281785975 >>>> oerrors 0 >>>> opackets 895533 >>>> rbytes 2062678 >>>> snaptime 189595.122347726 >>>> >>>> So this means the network only passed ~1.6 MB of data? >>> >>> is this on the outbound side or inbound? >>> >>> looks like it has a total of 8794120008 bits in the outbound side (obytes) >>> and 14036320 bits on the inbound (rbytes). What is the duration >>> 93 seconds? So the outbound would be ~94 Mbps and inbound 150 Kbps. >>> >> >> Sorry, outbound. I was looking at the wrong row. With the flow set at 50 >> I'm getting getting ~37 Mbps. In my mind the flow is not respecting the >> value I'm giving it. With the maxbw set to 100 I could get ~80 Mbps and now >> I'm only seeing ~37Mbps. Could it be something in the way Crossbow is >> averaging the actually bits and bytes going down the pipe or is it me not >> understanding the system? > > For TCP you might get a little less than the set bandwidth as TCP > would do flow control too (i.e if for bandwidth reasons ACKs come back > slower or packets get dropped, TCP will react and will slow down). Also, > note that the current property (maxbw) is a *limit* which means we will > ensure that the flow will not get *more* than what you have asked for. > Going forwards we are looking at adding guarantee where in the flow would > get what it asked for. > > -venu > >> >> root at myhost:/tmp# flowadm show-flowprop http-flow >> FLOW PROPERTY VALUE DEFAULT POSSIBLE >> http-flow maxbw 50 -- 50 >> http-flow priority medium -- medium >> >> root at myhost:/tmp# cat http-flow-* >> module: unix instance: 0 >> name: http-flow class: flow >> crtime 188662.418688816 >> ierrors 0 >> ipackets 32346 >> obytes 1300129135 >> oerrors 0 >> opackets 908351 >> rbytes 2133486 >> snaptime 191261.607310645 >> >> module: unix instance: 0 >> name: http-flow class: flow >> crtime 188662.418688816 >> ierrors 0 >> ipackets 85361 >> obytes 2399401106 >> oerrors 0 >> opackets 1676468 >> rbytes 5632604 >> snaptime 192621.873203347 >> >> >> [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat >> --13:52:20-- http://sol/myfile.dat >> Resolving sol... 192.168.69.104 >> Connecting to sol|192.168.69.104|:80... connected. >> HTTP request sent, awaiting response... 200 OK >> Length: 1048576000 (1000M) >> Saving to: `myfile.dat' >> >> 100%[==========================================================>] >> 1,048,576,000 4.06M/s in 3m 44s >> >> 13:56:04 (4.46 MB/s) - `myfile.dat' saved [1048576000/1048576000] >> >> >> real 3m44.364s >> user 0m1.042s >> sys 0m24.065s >> >> >> >>> >>>> >>>> The wget command was telling me it was getting ~80Mbps which is under the >>>> threshold of the flow. >>>> >>>> [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat >>>> --13:21:49-- http://sol/myfile.dat >>>> Resolving sol... 192.168.69.104 >>>> Connecting to sol|192.168.69.104|:80... connected. >>>> HTTP request sent, awaiting response... 200 OK >>>> Length: 1048576000 (1000M) >>>> Saving to: `myfile.dat' >>>> >>>> 100%[==========================================================>] >>>> 1,048,576,000 10.5M/s in 93s >>>> >>>> 13:23:23 (10.7 MB/s) - `myfile.dat' saved [1048576000/1048576000] >>>> >>>> >>>> real 1m33.701s >>>> user 0m0.899s >>>> sys 0m23.874s >>>> >>>> Maybe a value of 100Mbps is too high for this machine. I might try with >>>> 50Mbps to see what wget says. >>>> >>> >>> OK, >>> >>> -venu >>> >>>> -Cesar >>>> >>>> >>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> root at myhost:~# flowadm show-usage -s 11/23/2009,01:32:22 -e >>>>>>>> 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW" >>>>>>>> http-flow 01:32:22 01:32:42 1512 2571 0.001 Mbps >>>>>>>> ssh-flow 01:32:42 01:33:02 1818 3578 0.002 Mbps >>>>>>>> http-flow 01:33:02 01:33:22 66917 3165136 1.292 >>>>>>>> Mbp >>>>>>>> ssh-flow 01:33:02 01:33:22 3618 5344 0.003 Mbps >>>>>>>> http-flow 01:33:22 01:33:42 117947 5713018 2.332 >>>>>>>> Mbp >>>>>>>> ssh-flow 01:33:22 01:33:42 4182 3020 0.002 Mbps >>>>>>>> http-flow 01:33:42 01:34:02 118998 5685520 2.321 >>>>>>>> Mbp >>>>>>>> ssh-flow 01:33:42 01:34:02 11616 9924 0.008 Mbps >>>>>>>> http-flow 01:34:02 01:34:22 117084 5725664 2.337 >>>>>>>> Mbp >>>>>>>> http-flow 01:34:22 01:34:42 119130 5725168 2.337 >>>>>>>> Mbp >>>>>>>> http-flow 01:34:42 01:35:02 114180 5725168 2.335 >>>>>>>> Mbp >>>>>>>> http-flow 01:35:02 01:35:22 109230 5725664 2.333 >>>>>>>> Mbp >>>>>>>> http-flow 01:35:22 01:35:42 116160 5725168 2.336 >>>>>>>> Mbp >>>>>>>> http-flow 01:35:42 01:36:02 119262 5725168 2.337 >>>>>>>> Mbp >>>>>>>> http-flow 01:36:02 01:36:22 119196 5725664 2.337 >>>>>>>> Mbp >>>>>>>> http-flow 01:36:22 01:36:42 117216 5725168 2.336 >>>>>>>> Mbp >>>>>>>> http-flow 01:36:42 01:37:02 119394 5722636 2.336 >>>>>>>> Mbp >>>>>>>> http-flow 01:37:02 01:37:22 119526 5725168 2.337 >>>>>>>> Mbp >>>>>>>> http-flow 01:37:22 01:37:42 119460 5725168 2.337 >>>>>>>> Mbp >>>>>>>> http-flow 01:37:42 01:38:02 119460 5725664 2.338 >>>>>>>> Mbp >>>>>>>> http-flow 01:38:02 01:38:22 119724 5725168 2.337 >>>>>>>> Mbp >>>>>>>> http-flow 01:38:22 01:38:42 119724 5725168 2.337 >>>>>>>> Mbp >>>>>>>> http-flow 01:38:42 01:39:02 119130 5722636 2.336 >>>>>>>> Mbp >>>>>>>> http-flow 01:39:02 01:39:22 118866 5725168 2.337 >>>>>>>> Mbp >>>>>>>> http-flow 01:39:22 01:39:42 116490 5725664 2.336 >>>>>>>> Mbp >>>>>>>> http-flow 01:39:42 01:40:02 119790 5725168 2.337 >>>>>>>> Mbp >>>>>>>> http-flow 01:40:02 01:40:22 117678 5725168 2.337 >>>>>>>> Mbp >>>>>>>> http-flow 01:40:22 01:40:42 118668 5725664 2.337 >>>>>>>> Mbp >>>>>>>> http-flow 01:40:42 01:41:02 117414 5725168 2.337 >>>>>>>> Mbp >>>>>>>> http-flow 01:41:02 01:41:22 119790 5725168 2.337 >>>>>>>> Mbp >>>>>>>> http-flow 01:41:22 01:41:42 119813 5720510 2.336 >>>>>>>> Mbp >>>>>>>> http-flow 01:41:42 01:42:02 119394 5725664 2.338 >>>>>>>> Mbp >>>>>>>> http-flow 01:42:02 01:42:22 119724 5722272 2.336 >>>>>>>> Mbp >>>>>>>> http-flow 01:42:22 01:42:42 119526 5725664 2.338 >>>>>>>> Mbp >>>>>>>> http-flow 01:42:42 01:43:02 119196 5722140 2.336 >>>>>>>> Mbp >>>>>>>> http-flow 01:43:02 01:43:22 119394 5725664 2.338 >>>>>>>> Mbp >>>>>>>> http-flow 01:43:22 01:43:42 119658 5725168 2.337 >>>>>>>> Mbp >>>>>>>> http-flow 01:43:42 01:44:02 119064 5725168 2.337 >>>>>>>> Mbp >>>>>>>> http-flow 01:44:02 01:44:22 113256 5676668 2.315 >>>>>>>> Mbp >>>>>>>> ssh-flow 01:44:02 01:44:22 18414 49646 0.027 Mbps >>>>>>>> http-flow 01:44:22 01:44:42 118206 5725664 2.337 >>>>>>>> Mbp >>>>>>>> http-flow 01:44:42 01:45:02 117282 5722140 2.335 >>>>>>>> Mbp >>>>>>>> ssh-flow 01:44:42 01:45:02 4698 3544 0.003 Mbps >>>>>>>> http-flow 01:45:02 01:45:22 118536 5688284 2.322 >>>>>>>> Mbp >>>>>>>> ssh-flow 01:45:02 01:45:22 4092 3198 0.002 Mbps >>>>>>>> http-flow 01:45:22 01:45:42 119130 5725168 2.337 >>>>>>>> Mbp >>>>>>>> ssh-flow 01:45:22 01:45:42 1980 1478 0.001 Mbps >>>>>>>> >>>>>>>> That's above the flow's maxbw parameter. After that I tried to change >>>>>>>> the maxbw of the link with dladm and that brought the bandwidth down >>>>>>>> but still not down to 1.2 Mbps. >>>>>>>> >>>>>>>> root at myhost:~# dladm show-linkprop -p maxbw e1000g0 >>>>>>>> LINK PROPERTY PERM VALUE DEFAULT >>>>>>>> POSSIBLE >>>>>>>> e1000g0 maxbw rw 1.228 -- -- >>>>>>>> >>>>>>>> root at myhost:~# flowadm show-usage -s 11/23/2009,01:46:02 -e >>>>>>>> 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW" >>>>>>>> http-flow 01:46:02 01:46:22 119394 5725168 2.337 >>>>>>>> Mbp >>>>>>>> ssh-flow 01:46:02 01:46:22 4680 2980 0.003 Mbps >>>>>>>> http-flow 01:46:22 01:46:42 94314 4520316 1.845 >>>>>>>> Mbp >>>>>>>> >>>>>>>> Any ideas or is there a subtlety that I'm missing and the behavior is >>>>>>>> correct? >>>>>>>> >>>>>>>> Thanks for the help. >>>>>>>> >>>>>>>> -Cesar >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> crossbow-discuss mailing list >>>>>>>> crossbow-discuss at opensolaris.org >>>>>>>> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss >>>>>>>> >>>>>> >>>>>> >>>> >>>> >> >>
