Yes fat flow will help. You can also try increasing the default flow table size from 512 to 2M. And decreasing flow timeout from 3 minutes to 5seconds.
Regards -Harshad > On Jun 9, 2016, at 5:31 AM, Jacek Czerniak <jck.czern...@gmail.com> wrote: > > Hi All! > > We are testing LBaaS in OpenContrail 2.22.2 version and we saw very strange > behevior. > > > When we attach 1 vm to lbaas pool the performance is pretty good (it's about > 20k transaction/sec per LBaaS, it's almost equal pure vm without LBaaS). > > When we add second or third VM to the lbaas pool performance is decreasing > dramatically (it's about 10k transaction/sec per LBaaS) also the vrouter > flows is increasing about 5 times > > > Is it normal behevior? > > I find the commit for LBaaS VIP fat flow but only for master repo not for > 2.22 (https://bugs.launchpad.net/juniperopenstack/+bug/1583247) > > > I send my observations below. I will be grateful for your help! > > > > > LBaaS with 1 vm > > > Test > > siege -c 100 -b http://lbaas_ip > > > ** SIEGE 2.70 > ** Preparing 100 concurrent users for battle. > The server is now under siege... > Lifting the server siege... done. > Transactions: 13872683 hits > Availability: 99.19 % > Elapsed time: 744.60 secs > Data transferred: 476.28 MB > Response time: 0.01 secs > Transaction rate: 18631.05 trans/sec > Throughput: 0.64 MB/sec > Concurrency: 99.73 > Successful transactions: 13872683 > Failed transactions: 113078 > Longest transaction: 5.05 > Shortest transaction: 0.00 > > > flow -r > > New = 10554, Flow setup rate = 18982 flows/sec, Flow rate = 18982 flows/sec, > for last 556 ms > New = -46, Flow setup rate = -84 flows/sec, Flow rate = -84 flows/sec, for > last 546 ms > New = 64, Flow setup rate = 117 flows/sec, Flow rate = 117 flows/sec, for > last 546 ms > New = -34, Flow setup rate = -62 flows/sec, Flow rate = -62 flows/sec, for > last 546 ms > New = -3, Flow setup rate = -5 flows/sec, Flow rate = -5 flows/sec, for > last 546 ms > New = 106, Flow setup rate = 191 flows/sec, Flow rate = 191 flows/sec, for > last 553 ms > New = 64, Flow setup rate = 110 flows/sec, Flow rate = 110 flows/sec, for > last 579 ms > New = -8, Flow setup rate = -14 flows/sec, Flow rate = -14 flows/sec, for > last 563 ms > New = -13, Flow setup rate = -23 flows/sec, Flow rate = -23 flows/sec, for > last 545 ms > New = 73, Flow setup rate = 133 flows/sec, Flow rate = 133 flows/sec, for > last 545 ms > New = 51, Flow setup rate = 93 flows/sec, Flow rate = 93 flows/sec, for > last 547 ms > New = 6, Flow setup rate = 11 flows/sec, Flow rate = 12 flows/sec, for > last 545 ms > New = 44, Flow setup rate = 80 flows/sec, Flow rate = 78 flows/sec, for > last 546 ms > New = 157, Flow setup rate = 289 flows/sec, Flow rate = 289 flows/sec, for > last 543 ms > New = -28, Flow setup rate = -50 flows/sec, Flow rate = -50 flows/sec, for > last 554 ms > New = -18, Flow setup rate = -31 flows/sec, Flow rate = -31 flows/sec, for > last 577 ms > ... > > flow -s > > Flow Statistics > --------------- > Total Entries --- Total = 11576, new = 16 > Active Entries --- Total = 11576, new = 16 > Hold Entries --- Total = 0, new = 0 > Fwd flow Entries - Total = 11574 > drop flow Entries - Total = 2 > NAT flow Entries - Total = 0 > > Rate of change of Active Entries > -------------------------------- > current rate = 29 > Avg setup rate = 20823 > Avg teardown rate = 0 > Rate of change of Flow Entries > ------------------------------ > current rate = 29 > > > > dropstat > > NO - Flow Action Drop > > PKT0 int rate 380 pps > > Comments: > Vrouter create about 20k new flow per minute and aged about 20K flow per > minute. Vrouter CPU usage ~70%, haproxy usage ~90% > > > > > LBaaS with 2/3 vm's > > Test > > siege -c 100 -b http://lbaas_ip > > ** SIEGE 2.70 > ** Preparing 100 concurrent users for battle. > The server is now under siege... > > Lifting the server siege... done. > Transactions: 1832187 hits > Availability: 100.00 % > Elapsed time: 173.98 secs > Data transferred: 62.90 MB > Response time: 0.01 secs > Transaction rate: 10531.02 trans/sec > Throughput: 0.36 MB/sec > Concurrency: 99.47 > Successful transactions: 1832187 > Failed transactions: 0 > Longest transaction: 5.06 > Shortest transaction: 0.00 > > > flow -r > > New = 141242, Flow setup rate = 250429 flows/sec, Flow rate = 250508 > flows/sec, for last 564 ms > New = -713, Flow setup rate = -1287 flows/sec, Flow rate = -1250 flows/sec, > for last 554 ms > New = -66, Flow setup rate = -118 flows/sec, Flow rate = -215 flows/sec, for > last 556 ms > New = -194, Flow setup rate = -350 flows/sec, Flow rate = -225 flows/sec, for > last 554 ms > New = 766, Flow setup rate = 1377 flows/sec, Flow rate = 1278 flows/sec, for > last 556 ms > New = 165, Flow setup rate = 297 flows/sec, Flow rate = 389 flows/sec, for > last 554 ms > New = 641, Flow setup rate = 1152 flows/sec, Flow rate = 1057 flows/sec, for > last 556 ms > New = 210, Flow setup rate = 378 flows/sec, Flow rate = 436 flows/sec, for > last 555 ms > New = 801, Flow setup rate = 1443 flows/sec, Flow rate = 1405 flows/sec, for > last 555 ms > New = -488, Flow setup rate = -879 flows/sec, Flow rate = -834 flows/sec, for > last 555 ms > New = -433, Flow setup rate = -781 flows/sec, Flow rate = -837 flows/sec, for > last 554 ms > New = 287, Flow setup rate = 516 flows/sec, Flow rate = 541 flows/sec, for > last 556 ms > New = 370, Flow setup rate = 666 flows/sec, Flow rate = 690 flows/sec, for > last 555 ms > ... > > flow -s > > Flow Statistics > --------------- > Total Entries --- Total = 146402, new = 574 > Active Entries --- Total = 146360, new = 559 > Hold Entries --- Total = 42, new = 15 > Fwd flow Entries - Total = 146358 > drop flow Entries - Total = 2 > NAT flow Entries - Total = 0 > > Rate of change of Active Entries > -------------------------------- > current rate = 1001 > Avg setup rate = 130994 > Avg teardown rate = 0 > Rate of change of Flow Entries > ------------------------------ > current rate = 1028 > > > dropstat > > Flow Action Drop 40092 (increasing) > > PKT0 interface rate 4.46K > > Comments: > Vrouter create about 400k new flow per minute and aged about 400K flow per > minute. It's droppig about 300 flow per minute (flow_action_drop). Vrouter > CPU usage - 420%, haproxy CPU usage ~90% > > > > > -- > Best Regards, > Jacek Czerniak > _______________________________________________ > Users mailing list > Users@lists.opencontrail.org > http://lists.opencontrail.org/mailman/listinfo/users_lists.opencontrail.org
_______________________________________________ Users mailing list Users@lists.opencontrail.org http://lists.opencontrail.org/mailman/listinfo/users_lists.opencontrail.org
