Hi All! We are testing LBaaS in OpenContrail 2.22.2 version and we saw very strange behevior.
When we attach 1 vm to lbaas pool the performance is pretty good (it's about 20k transaction/sec per LBaaS, it's almost equal pure vm without LBaaS). When we add second or third VM to the lbaas pool performance is decreasing dramatically (it's about 10k transaction/sec per LBaaS) also the vrouter flows is increasing about 5 times Is it normal behevior? I find the commit for LBaaS VIP fat flow but only for master repo not for 2.22 (https://bugs.launchpad.net/juniperopenstack/+bug/1583247) I send my observations below. I will be grateful for your help! LBaaS with 1 vm Test siege -c 100 -b http://lbaas_ip ** SIEGE 2.70 ** Preparing 100 concurrent users for battle. The server is now under siege... Lifting the server siege... done. Transactions: 13872683 hits Availability: 99.19 % Elapsed time: 744.60 secs Data transferred: 476.28 MB Response time: 0.01 secs Transaction rate: 18631.05 trans/sec Throughput: 0.64 MB/sec Concurrency: 99.73 Successful transactions: 13872683 Failed transactions: 113078 Longest transaction: 5.05 Shortest transaction: 0.00 flow -r New = 10554, Flow setup rate = 18982 flows/sec, Flow rate = 18982 flows/sec, for last 556 ms New = -46, Flow setup rate = -84 flows/sec, Flow rate = -84 flows/sec, for last 546 ms New = 64, Flow setup rate = 117 flows/sec, Flow rate = 117 flows/sec, for last 546 ms New = -34, Flow setup rate = -62 flows/sec, Flow rate = -62 flows/sec, for last 546 ms New = -3, Flow setup rate = -5 flows/sec, Flow rate = -5 flows/sec, for last 546 ms New = 106, Flow setup rate = 191 flows/sec, Flow rate = 191 flows/sec, for last 553 ms New = 64, Flow setup rate = 110 flows/sec, Flow rate = 110 flows/sec, for last 579 ms New = -8, Flow setup rate = -14 flows/sec, Flow rate = -14 flows/sec, for last 563 ms New = -13, Flow setup rate = -23 flows/sec, Flow rate = -23 flows/sec, for last 545 ms New = 73, Flow setup rate = 133 flows/sec, Flow rate = 133 flows/sec, for last 545 ms New = 51, Flow setup rate = 93 flows/sec, Flow rate = 93 flows/sec, for last 547 ms New = 6, Flow setup rate = 11 flows/sec, Flow rate = 12 flows/sec, for last 545 ms New = 44, Flow setup rate = 80 flows/sec, Flow rate = 78 flows/sec, for last 546 ms New = 157, Flow setup rate = 289 flows/sec, Flow rate = 289 flows/sec, for last 543 ms New = -28, Flow setup rate = -50 flows/sec, Flow rate = -50 flows/sec, for last 554 ms New = -18, Flow setup rate = -31 flows/sec, Flow rate = -31 flows/sec, for last 577 ms ... flow -s Flow Statistics --------------- Total Entries --- Total = 11576, new = 16 Active Entries --- Total = 11576, new = 16 Hold Entries --- Total = 0, new = 0 Fwd flow Entries - Total = 11574 drop flow Entries - Total = 2 NAT flow Entries - Total = 0 Rate of change of Active Entries -------------------------------- current rate = 29 Avg setup rate = 20823 Avg teardown rate = 0 Rate of change of Flow Entries ------------------------------ current rate = 29 dropstat NO - Flow Action Drop PKT0 int rate 380 pps Comments: Vrouter create about 20k new flow per minute and aged about 20K flow per minute. Vrouter CPU usage ~70%, haproxy usage ~90% LBaaS with 2/3 vm's Test siege -c 100 -b http://lbaas_ip ** SIEGE 2.70 ** Preparing 100 concurrent users for battle. The server is now under siege... Lifting the server siege... done. Transactions: 1832187 hits Availability: 100.00 % Elapsed time: 173.98 secs Data transferred: 62.90 MB Response time: 0.01 secs Transaction rate: 10531.02 trans/sec Throughput: 0.36 MB/sec Concurrency: 99.47 Successful transactions: 1832187 Failed transactions: 0 Longest transaction: 5.06 Shortest transaction: 0.00 flow -r New = 141242, Flow setup rate = 250429 flows/sec, Flow rate = 250508 flows/sec, for last 564 ms New = -713, Flow setup rate = -1287 flows/sec, Flow rate = -1250 flows/sec, for last 554 ms New = -66, Flow setup rate = -118 flows/sec, Flow rate = -215 flows/sec, for last 556 ms New = -194, Flow setup rate = -350 flows/sec, Flow rate = -225 flows/sec, for last 554 ms New = 766, Flow setup rate = 1377 flows/sec, Flow rate = 1278 flows/sec, for last 556 ms New = 165, Flow setup rate = 297 flows/sec, Flow rate = 389 flows/sec, for last 554 ms New = 641, Flow setup rate = 1152 flows/sec, Flow rate = 1057 flows/sec, for last 556 ms New = 210, Flow setup rate = 378 flows/sec, Flow rate = 436 flows/sec, for last 555 ms New = 801, Flow setup rate = 1443 flows/sec, Flow rate = 1405 flows/sec, for last 555 ms New = -488, Flow setup rate = -879 flows/sec, Flow rate = -834 flows/sec, for last 555 ms New = -433, Flow setup rate = -781 flows/sec, Flow rate = -837 flows/sec, for last 554 ms New = 287, Flow setup rate = 516 flows/sec, Flow rate = 541 flows/sec, for last 556 ms New = 370, Flow setup rate = 666 flows/sec, Flow rate = 690 flows/sec, for last 555 ms ... flow -s Flow Statistics --------------- Total Entries --- Total = 146402, new = 574 Active Entries --- Total = 146360, new = 559 Hold Entries --- Total = 42, new = 15 Fwd flow Entries - Total = 146358 drop flow Entries - Total = 2 NAT flow Entries - Total = 0 Rate of change of Active Entries -------------------------------- current rate = 1001 Avg setup rate = 130994 Avg teardown rate = 0 Rate of change of Flow Entries ------------------------------ current rate = 1028 dropstat Flow Action Drop 40092 (increasing) PKT0 interface rate 4.46K Comments: Vrouter create about 400k new flow per minute and aged about 400K flow per minute. It's droppig about 300 flow per minute (flow_action_drop). Vrouter CPU usage - 420%, haproxy CPU usage ~90% -- Best Regards, Jacek Czerniak
_______________________________________________ Users mailing list [email protected] http://lists.opencontrail.org/mailman/listinfo/users_lists.opencontrail.org
